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ABSTRACT 


Many  military  Intelligence  Surveillance  and  Reconnaissance  (ISR) 
operations  would  benefit  greatly  from  a  fleet  of  disparate  sensor-bearing  UAVs 
that  are  tightly  integrated  via  a  communications  network,  work  cooperatively  for  a 
common  operational  objective,  enhance  situation  awareness  of  the  areas  of 
operation,  and  increase  persistence  of  sensor  dwell  time  on  strategic  targets. 
This  would  enable  continuity  in  the  entire  target  acquisition  cycle,  from  detection 
to  classification  to  identification  and  finally  localization  of  targets,  in  a  diverse  and 
dynamic  environment.  The  integration  of  sensors  and  development  of  tactics  in  a 
cooperative  sensing  environment  is  one  of  the  current  focuses  among  the  military 
intelligence  community,  and  hence  motivates  this  thesis  effort.  By  building 
models  with  an  existing  agent-based  simulation  platform  and  using  an  extremely 
efficient  experimental  design  methodology,  numerous  factors  which  could 
potentially  affect  the  effectiveness  of  a  cooperative  sensing  network  against  two 
arrays  of  targets  are  explored.  The  factors  considered  include  UAV  airspeed, 
reliability,  detection/classification  coverage  and  probability,  network  latency  and 
degradation,  UAV  configurations  and  responsiveness,  as  well  as  air  space 
separation.  The  two  arrays  of  targets  are  mobile  armor  concentrations  and  time 
critical  targets;  these  vary  in  their  deployment  profiles,  vulnerability  constraints 
and  ease  of  detectability.  Factors  characterizing  these  targets,  such  as  the  shoot- 
and-scoot  behavior  of  time  critical  targets,  are  also  investigated.  The  study 
provides  operational  insights  pertaining  to  the  design  and  effective  use  of 
cooperative  sensing  for  ISR  purposes.  These  include  the  importance  of  having 
good  UAV  sensor  capabilities,  the  need  for  a  suite  of  sensors  to  aid  in  locating 
well-camouflaged  time-critical  targets,  and  the  need  for  “intelligent”  application  of 
UAV  cooperation  tactics  based  on  the  characteristics  of  recently-classified 
targets. 
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THESIS  DISCLAIMER 


The  reader  is  cautioned  that  computer  programs  developed  in  this 
research  may  not  have  been  exercised  for  all  cases  of  interest.  While  every  effort 
has  been  made,  within  the  time  available,  to  ensure  that  the  programs  are  free  of 
computational  and  logic  errors,  they  cannot  be  considered  validated.  Any 
application  of  these  programs  without  additional  verification  is  at  the  risk  of  the 
user. 
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EXECUTIVE  SUMMARY 


As  a  result  of  combat  operations  during  Operation  Iraqi  Freedom,  a  major 
lesson  learned  indicates  that  the  success  of  a  battle  is  increasingly  reliant  on 
more  accurate  and  timely  collection  of  the  battle-space  intelligence.  Unmanned 
systems,  and  certainly  Unmanned  Aerial  Vehicles  (UAVs),  are  widely  used  in 
today’s  military  ISR  operations,  and  will  be  even  more  prevalent  in  the  next 
generation  military  as  warfighting  concepts  such  as  Network  Centric  Warfare  and 
Distributed  Operation  emerge.  A  number  of  significant  military  Intelligence 
Surveillance  and  Reconnaissance  (ISR)  operations  would  benefit  greatly  from  a 
fleet  of  disparate  sensor-bearing  UAVs  that  are  integrated  via  a  communications 
network,  work  in  a  cooperative  manner  for  a  common  operational  objective, 
enhance  situation  awareness  of  the  areas  of  operation,  and  increase  persistence 
of  sensor  time  on  strategic  targets.  Furthermore,  the  cooperative  sensing 
network  enables  continuity  in  the  entire  target  acquisition  cycle,  from  detection  to 
classification  to  identification  and  finally  localization  of  interested  targets,  in  a 
diverse  and  dynamic  environment.  The  integration  and  tactics  development  in  a 
cooperative  sensing  environment  are  currently  key  focuses  among  the  military 
intelligence  community,  and  serve  as  the  motivation  for  this  thesis. 

The  objective  of  the  thesis  is  to  explore  the  enhancement  in  detection  and 
classification  capabilities  for  mission  critical  and  time  critical  targets,  should  the 
capability  of  a  cooperative  sensing  network  becomes  available  to  the  Marine 
Expeditionary  Brigade  commander.  Two  types  of  UAVs  are  considered:  the  Low 
Altitude  and  Endurance  (LAE)  and  the  Medium  Altitude  and  Endurance  (MAE). 
The  two  target  types  in  the  study  include  the  time  critical  targets  and  the  mobile 
armor  concentrations;  they  represent  the  two  main  arrays  of  targets 
characterized  by  their  deployment  profiles,  vulnerability  constraints  and  ease  of 
detectability.  The  study  measures  the  proportion  of  the  enemy’s  mobile  targets 
and  Time  Critical  Targets  (TCTs)  classified  over  a  four-hour  period  for 
Intelligence  Preparation  of  the  Battlefield  prior  to  a  Marine  Expeditionary 

xvii 


amphibious  assault.  In  doing  so,  the  author  applies  suitable  analytical 
methodologies  to  gain  insights  into  questions  as  these: 

•  What  are  good  cooperative  sensing  network  configurations? 

•  Are  there  diminishing  returns  with  increasing  UAVs  allocated? 

•  What  are  good  UAV  responses  in  a  cooperative  sensing  environment? 

•  What  are  the  effects  of  degraded  communications  and  increasing 
latency? 

•  What  are  the  effects  of  UAV  failure  in  the  presence  of  cooperative 
sensing? 

A  number  of  models  are  developed  with  reference  to  the  Marine  Corps 
Warfighting  Laboratory’s  Sea  Viking  2004  scenario  using  the  agent-based 
simulation  platform  Map  Aware  Non-uniform  Automata  (MANA).  The  computing 
resource  requirements  for  these  models  are  much  lower  than  for  most 
conventional  simulation  models,  and  can  be  executed  over  tens  of  thousands 
repetitions  to  explore  many  possible  representations  of  real-life  situations  Time 
and  cost  considerations  would  preclude  running  even  a  fraction  of  these 
situations  in  a  field  experiment,  so  the  simulation  results  provide  useful  insights 
regarding  overall  system  effectiveness.  A  total  of  20  factors  varying  over  1290 
design  points  are  considered.  Controllable  factors  pertaining  to  the  number  of 
LAEs,  UAV  characteristics  such  as  airspeed,  detection  range,  classification 
range  and  probability  of  classification,  network  parameters  including  link  reliability 
and  message  latency,  as  well  as  air-space  separation  are  explored.  Factors 
uncontrollable  in  real-world  settings,  such  as  enemy  concealment,  counter 
detection  sensor  range  and  period  of  vulnerability,  are  also  varied  in  the 
simulation  runs.  The  1290  excursions  with  50  repetitions  each,  amounting  to 
64,500  MANA  runs,  are  submitted  to  the  MAUI  High  Performance  Computing 
Center  for  computation.  Another  set  of  models  and  excursions  are  also 
generated  to  study  the  effects  of  an  LAE  failure  at  specific  time  periods  during 
the  four-hour  operation. 
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The  data  collected  from  the  simulations  is  analyzed  using  several 
graphical  and  statistical  tools.  Multiple  Linear  Regression  and  Data  Partitioning 
are  used  in  conjunction  with  the  results  from  other  statistical  techniques  to  draw 
useful  insights  and  operational  guidance  relevant  to  the  employment  of 
cooperative  sensing  for  military  ISR  purposes. 

The  results  show  approximately  9.8%  of  the  TCTs  and  27.2%  of  the  armor 
targets  are  classified  with  the  configuration  of  an  MAE  and  four  LAEs  at  the  end 
of  the  four-hour  operation.  The  relatively  low  classification  measures  are 
attributed  to  the  relatively  small  coverage  area  extended  throughout  the  entire 
duration  of  operation  as  compared  to  the  considerably  large  area  of  interest.  The 
analysis  leads  to  the  observation  that  the  classification  effectiveness  depends  on 
the  UAV  routings,  and  is  an  important  consideration  when  preparing  an 
intelligence  collection  plan.  Prior  knowledge  of  the  enemy’s  courses  of  action, 
target’s  profiles  and  terrain  information  is  beneficial.  An  LAE  should  be  more 
responsive  to  the  MAE  cueing  to  achieve  higher  classification  performance 
against  TCTs  and  less  responsive  to  the  MAE  cueing  when  acquiring  armor 
targets.  However,  an  LAE  should  tend  to  follow  a  new  classified  enemy  contact 
when  searching  out  armor  targets  as  they  are  generally  deployed  in  formations. 

While  much  is  discussed  about  the  behaviors  of  the  UAVs  in  a  cooperative 
sensing  environment  with  shared  situation  awareness,  many  fundamental  sensor 
characteristics  like  longer  classification  range  and  higher  classification  probability 
also  have  impacts  on  the  overall  classification  effectiveness.  However,  an  LAE 
classification  range  that  is  larger  than  5  km  easily  distracts  the  LAE;  more  clutter 
appears  in  the  sensor  view  and  penalizes  the  overall  classification  performance. 
An  increase  in  number  of  LAEs  generally  improves  the  target  classification 
proportions,  but  the  proportion  of  armor  classification  seems  to  taper  off  at  four 
LAEs  for  this  scenario. 

Link  reliability  and  message  latency  are  shown  to  have  no  significant 
effects  on  the  classification  outcome.  Air-space  separations  also  do  not  matter. 
High  TCT  concealment  capabilities  and  counter-detection  sensor  ranges 


significantly  hinder  good  classification  performance  of  the  sensing  network.  A 
UAV  equipped  solely  with  an  electro-optics  sensor  may  not  be  effective  against 
the  well-camouflage  TCTs.  Against  such  targets,  other  sensors  or  radars  may  be 
a  more  viable  option. 

There  is  some  indication  showing  that  cooperative  sensing  does  provide  a 
more  robust  solution  in  some  scenarios  in  terms  of  the  classification  performance 
should  an  LAE  fail  or  be  shot  down  during  the  four-hour  operation.  Following  a 
fixed  set  of  standard  operating  tactics  regardless  of  the  targets  of  interest  and 
enemy’s  courses  of  action  is  not  the  way  forward  in  employing  a  cooperative 
sensing  network. 


xx 


I.  INTRODUCTION 


A.  OVERVIEW 

Unmanned  Aerial  Vehicles  (UAVs)  are  remotely  piloted  or  self-piloted 
aircraft  that  carry  cameras,  sensors,  communication  equipment  or  other 
payloads.  They  have  been  used  primarily  in  Intelligence,  Surveillance  and 
Reconnaissance  (ISR)  roles  since  the  1950s.  Since  1964  the  Department  of 
Defense  (DoD),  having  realized  the  potential  of  UAVs  as  a  key  component  in  ISR 
missions,  had  11  different  types  of  UAV  developed.  However,  due  to  acquisition 
and  development  problems,  only  three  entered  production.  The  U.S.  Navy  has 
also  studied  the  feasibility  of  operating  Vertical  Takeoff  and  Landing  (VTOL) 
UAVs  since  the  early  1960s  to  overcome  the  limitation  of  short  takeoff  distance 
from  surface  vessels,  the  QH-50  Gyrodyne  torpedo-delivery  drone  being  an  early 
example.  Nonetheless,  high  cost  and  technological  immaturity  precluded 
acquiring  and  fielding  operational  VTOL  UAV  systems  at  that  time.  By  the  early 
1990s,  DoD  sought  UAVs  capable  of  satisfying  surveillance  requirements  in 
Close  Range,  Short  Range  or  Endurance  categories,  where  persistence, 
dynamic  re-tasking  capability  and  real-time  imagery  are  critical  to  the  mission.  By 
the  late  1990s,  the  Close  and  Short  Range  categories  were  combined,  and  a 
separate  Shipboard  category  emerged.  The  current  classes  of  these  vehicles  are 
the  Tactical  UAV  and  the  Long  Endurance  UAV,  however,  both  classes  are  still 
predominately  employed  for  ISR  purposes  (Federation  of  American  Scientists 
Intelligence  Resource  Program  website,  2005). 

The  use  of  UAVs  evolved  rapidly  over  the  last  two  decades  to  conduct  air- 
to-ground  strike  missions  with  the  development  of  the  Unmanned  Combat  Aerial 
Vehicles  (UCAVs).  The  operational  UCAV  system  is  envisioned  as  a  force 
enabler  for  the  U.S.  Air  Force  that  will  conduct  Suppression  of  Enemy  Air 
Defense  (SEAD)  and  strike  missions  in  support  of  post-2010  manned  strike 
packages.  This  SEAD/Strike  mission  will  be  the  first  instantiation  of  an  UCAV 

vision  that  will  evolve  into  a  broader  range  of  combat  missions  as  the  concept 
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and  technologies  mature,  and  the  UCAV  affordability  potential  is  realized.  The 
DARPA/Air  Force/Boeing  X-45A  technology  demonstration  aircraft  completed  its 
first  flight  on  22  May  2002.  Multi-aircraft  testing  began  in  2003  when  a  second  X- 
45A  became  operational,  leading  to  joint  UCAV  and  manned  exercises  in  2006 
(Federation  of  American  Scientists  Military  Analysis  Network  website,  2005).  As 
technology  advances  and  the  importance  of  unmanned  vehicles  as  a  force 
enabler  is  realized  by  the  military,  the  Stealth  UAV  becomes  the  current  class  of 
UAV  to  be  researched  and  developed. 

Being  an  unmanned  platform  and  generally  smaller  in  size,  the  UAV  is 
preferred  over  manned  aircraft  for  strategic  deployment  into  the  airspace  deep 
behind  the  adversary  line  of  operations  which  is  often  protected  by  heavy  air 
defense  elements.  Today  UAVs  are  widely  deployed  in  Time  Critical  Targeting 
(TCT)  operations,  where  long  endurance  and  dynamic  re-tasking  capabilities  are 
imperative  to  the  success  of  acquiring  of  high  value  targets  and  eventually 
executing  the  kill  chain  in  prosecuting  these  targets.  The  concept  of  cooperative 
sensing  soon  evolved  to  offer  the  ability  to  cue  available  and  suitable  joint 
sensors  networked  together  in  the  theatre  of  war  to  extend  the  persistence  of 
sensor  time  on  target.  This  is  currently  drawing  great  attention  of  military 
strategists  and  technologists.  Medium  Altitude  Endurance  (MAE)  UAVs  have  the 
capability  of  extended  flight  duration,  typically  6-12  hours  or  longer.  These  are 
supplemented  with  the  tactical  Low  Altitude  Endurance  (LAE)  UAVs  which 
enable  near-real-time  imagery  and  local  tracking  of  targets.  While  the  MAE  UAVs 
have  long-range  deployment  and  wide-area  surveillance  or  long  sensor  dwell 
over  the  target  area  to  provide  initial  detections,  the  close-in  LAE  UAVs  allow  the 
identification,  localization  and  tracking  of  interested  targets  with  high  accuracy. 

Today  there  are  some  50  U.S.  companies,  academic  institutions,  and 
government  organizations  developing  over  150  UAV  designs.  Forty  of  these 
companies  have  some  115  of  these  designs  flying,  i.e.,  at  least  one  working 
prototype  built.  Fifteen  of  these  companies  have  26  models  of  UAVs  in,  or  ready 
for,  production  (UAV  Forum  website,  2005).  With  recent  emphasis  in  Network 
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Centric  Warfare  (NCW)  which  extensively  exploits  UAVs  as  an  indispensable 
part  of  the  sensor  grid,  the  development  of  UAVs  will  definitely  be  an  important 
part  of  military  development  and  a  future  force  multiplier. 

B.  MOTIVATION 

As  a  result  of  combat  operations  during  Operation  Iraqi  Freedom,  a  major 
lesson  learned  indicates  that  a  better  asset  to  collect  battle-space  intelligence  is 
crucial  to  the  way  forward  (U.S.  Joint  Forces  Command,  2004).  A  number  of 
significant  military  operations  would  benefit  greatly  from  a  fleet  of  disparate 
sensor-bearing  UAVs  that  are  integrated  via  a  communications  network,  work  in 
a  cooperative  manner  for  a  common  operational  objective,  enhance  situation 
awareness  of  the  areas  of  operation,  and  increase  persistence  of  sensor  time  on 
strategic  targets.  Figure  1  depicts  a  cooperative  sensing  network  employed  in  a 
Time  Critical  Targeting  operation.  Multiple  UAVs  allow  sensing  to  be  performed 
in  parallel,  thereby  reducing  the  amount  of  time  required  to  gather  data.  If  a 
vehicle  becomes  disabled,  the  remaining  vehicles  can  continue  sensing, 
although  at  a  reduced  collection  rate.  Most  importantly,  the  cooperative  sensing 
network  enables  continuity  in  the  entire  target  acquisition  cycle,  from  detection  to 
classification  to  identification  and  finally  localization  of  interested  targets,  in  a 
diverse  and  dynamic  environment.  The  success  of  cooperative  sensing  depends 
significantly  on  two  key  technological  emphases:  the  development  of  state-of-art 
imaging  sensors  and  the  integration  of  sensor  and  ISR  systems  in  a  seamless 
and  cooperative  manner.  The  ensuing  paragraphs  provide  further  discussion  on 
these  technologies. 

Perhaps  the  need  for  a  complementary  suite  of  sensors  is  of  greater 

importance  for  surveillance  on  land  than  for  a  maritime  environment.  The 

richness  of  terrain  variation  and  coverage  on  the  ground  mean  that  targets 

moving  across  land  surfaces  could  employ  effective  Concealment,  Camouflage 

and  Deception  (CCD)  tactics  to  prevent  being  detected  by  the  adversary.  The 

entire  battle  of  ground  search  and  detection  revolves  around  the  ability  to  employ 
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sensing  equipments  and  technologies  that  are  superior  to  the  adversary’s  CCD 
capability.  Over  the  last  two  decades,  the  development  of  air-to-ground  sensor 
technologies  has  improved  by  leaps  and  bounds.  Conventional  microwave  radar 
is  now  able  to  detect  longer  and  perform  better  in  a  wider  range  of  operating 
environments.  Many  Electro-Optics  (EO)  and  Infra-Red  (IR)  imaging  sensors  can 
achieve  astonishing  spatial  resolutions  of  less  than  one  meter,  the  Foliage 
Penetration  (FOPEN)  Radar  and  Synthetic  Aperture  Radar  (SAR)  allow  detection 
of  targets  hidden  under  concealment  and  camouflage,  Laser  Radar  (LADAR) 
imaging  is  achievable  with  higher  spatial  resolution  and  precision  up  a  tenth  of  a 
meter,  and  the  “through-the-wall”  imaging  radar  is  under  research  and 
development  to  provide  the  capability  to  look  into  buildings  at  standoff  distances 
and  build  a  picture  of  the  tactical  situation.  The  availability  of  a  diverse  range  of 
air-to-ground  radars  and  imaging  sensors  makes  it  possible  for  a  fleet  of  UAVs, 
each  mounted  with  disparate  sensors,  to  be  integrated  to  form  a  cooperative 
sensing  network  to  collect  data  in  a  parallel,  coordinated  and  optimal  manner. 


Figure  1 .  A  Cooperative  Sensing  Network  Deployed  for  Time  Critical  Targeting 

Operation  [Best  Viewed  in  Color] 
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The  greater  challenge  of  cooperative  sensing,  in  the  author’s  opinion, 
does  not  lie  on  the  employment  of  advanced  sensors  as  one  could  expect  these 
sensors  to  be  widely  available  in  the  near  future  given  the  amazing  rate  at  which 
technology  advances.  The  greater  challenge  involves  seamless  integration  of  all 
the  sensors  in  the  theatre  of  operations  and  tactics  development  to  provide  high- 
quality,  uninterrupted  information  over  the  entire  target  life  to  the  commander  for 
making  battle  decisions  effectively.  This  involves  an  intelligent  sensing  system 
that  is  able  to  efficiently  task  suitable  and  available  sensors  on-the-fly  given 
various  tactical  situations.  The  sensing  system  should  also  be  capable  of 
prioritizing  conflicting  sensing  requirements,  if  any  should  occur.  The 
development  of  such  a  smart  system  is  by  no  means  trivial;  it  is  not  uncommon  to 
have  overwhelming  ISR  requests  beyond  the  availability  of  the  sensors.  The  tight 
integration  amongst  sensor  systems  has  prompted  the  need  for  developing  a 
high  throughput  and  robust  communications  network.  Obviously,  the  availability 
of  such  a  sensing  network  would  not  be  effective  without  restructuring  the  military 
ISR  processes,  just  as  adopting  the  Network  Centric  Warfare  paradigm  would  not 
be  effective  without  changing  the  traditional  ways  of  fighting  a  war. 


C.  BACKGROUND  WORK 

There  have  been  a  number  of  preceding  theses  at  the  Naval  Postgraduate 
School  (NPS)  that  use  an  agent-based  modeling  approach  to  explore  the  use  of 
UAVs  for  military  intelligence  gathering  missions.  Each  thesis  focuses  on  a 
different  aspect  of  using  UAVs  on  a  targeted  ISR  operational  scenario.  Below  are 
brief  summaries  of  some  of  these  works. 

Raffetto  (2004)  analyzes  the  impact  various  UAV  capabilities  have  on 

intelligence  gathering  missions  for  a  Marine  Expeditionary  Brigade  (MEB) 

commander’s  2015  UAV  to  support  rapid  planning  and  decision  making  for 

multiple  concurrent  operations.  This  facilitates  maneuver  and  precision 

engagement  based  on  the  UAV  operations  in  the  Sea  Viking  2004  (SV04) 

scenario  provided  by  Marine  Corps  Warfighting  Lab  (MCWL).  He  explores  the 
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validity  of  current  requirements  and  provides  insights  into  the  importance  of 
various  UAV  characteristics,  such  as  airspeed,  endurance,  sweep  width,  and 
sensor  capability  for  a  single  UAV.  The  analysis  seeks  to  measure  the  proportion 
of  the  enemy’s  entities  detected  over  time,  hence  the  ability  to  gain  superiority  in 
situation  awareness  (Raffetto,  2004). 

Berner  (2004)  explores  the  effective  use  of  multiple  UAVs  for  the  Navy’s 
Surface  Search  and  Control  mission  based  on  a  coastal  environment  scenario 
with  dense  shipping  traffic  and  sparse  enemy  contacts.  In  addition  to  the  impacts 
various  UAV  parameters  have  on  the  detection  and  classification  effectiveness  of 
the  enemy  contacts,  he  also  shows  the  ISR  effectiveness  for  different  UAV 
tactics  and  combinations  (Berner,  2004).  However,  although  the  contacts 
acquired  by  each  UAV  are  routed  to  a  central  command,  this  has  no  influence  on 
the  collection  plan  of  the  other  UAVs  as  would  occur  within  a  cooperative  sensing 
network. 

McMindes  (2005)  looks  at  how  UAV  survivability  is  affected  by  UAV 
speed,  stealth,  altitude,  and  sensor  range,  as  well  as  enemy  force  sensor  ranges, 
probability  of  kill,  array  of  forces,  and  numerical  strength.  The  analysis,  which  is 
based  upon  the  SV04  scenario  of  Raffetto  (2004),  concludes  that  a  UAV  is  highly 
survivable  with  fast  speed,  with  the  exception  that  stealth  becomes  more 
important  than  speed  alone  when  the  enemy  has  extremely  high-capability 
assets  (McMindes,  2005). 


D.  THE  PROBLEM 

The  Sea  Viking  Division  of  the  MCWL  is  responsible  for  experiment 
planning  and  design  of  the  ongoing  Sea  Viking  live  experimentation  campaign. 
Sea  Viking  is  planned  and  executed  in  two-year  segments.  The  current  focus  for 
Sea  Viking  2006  (SV06)  is  to  conduct  live  experimentation  that  will  assist  in  the 
development  of  operational  maneuver  from  the  sea/ship  to  objective  maneuver 
tactics,  techniques  and  procedures  and  to  develop  and  assess  experimental 
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capabilities  that  support  these  concepts.  Additionally,  SV06  will  examine  the 
emerging  Distributed  Operations  (DO)  concept  which  will  deploy  for  operations  in 
the  Global  War  on  Terror  (Marine  Corps  Warfighting  Laboratory  Sea  Viking 
website,  2005).  General  Hagee,  Commandant  of  the  Marine  Corps,  highlights  the 
required  capabilities  for  enhanced  intelligence  capabilities  to  collect,  report,  and 
exploit  intelligence  for  small  units  employing  DO.  These  include  employment  or 
direction  of  unmanned  ground  or  air  vehicles,  or  the  ability  to  access  command 
and  control  networks  for  the  purpose  of  extracting  specific  intelligence  pertinent 
to  the  unit’s  local  situation  (Hagee,  2005).  A  cooperative  sensing  capability  will 
indeed  provide  the  required  strategic  and  tactical  ISR  requirements  for  the 
emerging  DO  concept. 

The  objective  of  this  study  is  to  explore  the  enhancement  in  the 
effectiveness  on  detection  and  classification  capabilities  of  mission  critical  and 
time  critical  targets,  should  the  capability  of  a  cooperative  sensing  network 
become  available  to  the  MEB  commander.  These  targets  are  deemed  to  have 
significant  implications  for  subsequent  battle  developments,  and  hence  warrant 
elaborate  sensing  requirements  directed  toward  them.  The  study  aims  to 
measure  the  proportion  of  the  enemy’s  mobile  and  time  critical  targets  classified 
over  a  four-hour  period  for  Intelligence  Preparation  of  the  Battlefield  (IPB)  prior  to 
an  amphibious  assault,  thus  offering  insights  about  the  effectiveness  of  the 
cooperative  sensing  network  for  ISR  missions  against  these  target  types. 


E.  SCOPE  AND  ASSUMPTIONS 

The  research  focuses  primarily  on  formulating  and  building  models  to 

capture  essential  characteristics  of  UAVs  in  an  IPB  operation.  These  models  are 

explored  extensively  to  uncover  how  the  effectiveness  of  cooperative  sensing 

capability  is  affected  by  ISR  processes  and  structures  as  well  as  UAV  responses 

to  shared  Situation  Awareness  (SA).  An  existing  agent-based  distillation  called 

Map  Aware  Non-uniform  Automata  (MANA)  is  used  to  construct  the  scenario. 

The  scenario  is  modified  from  Raffetto’s  SV04  base  scenario  to  eliminate  the  red 
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infantry  elements  and  include  high  value  targets  such  as  the  Air  Defense  Artillery 
(ADA)  elements,  Surface-to-Air  Missile  (SAM)  launchers,  armor  concentrations 
and  Multiple  Launched  Rocket  Systems  (MLRS).  To  further  evaluate  the  effects 
of  cooperative  sensing,  communications  models  are  incorporated  into  the 
network  of  UAVs,  and  various  mixes  of  MAE  and  LAE  UAVs  are  introduced. 

The  model  exploration  uses  a  very  efficient  experimental  design 
methodology  to  capture  a  large  number  of  factors  and  their  interactions  which 
potentially  affect  the  scenario  outcomes.  The  controllable  factors  to  be 
considered  include  UAV  parameters  (airspeed,  reliability,  detection/classification 
coverage  and  probability),  network  parameters  (latency  and  degradation),  ISR 
configurations  (number  of  LAEs  to  one  MAE),  UAV  behaviors  (reactivity  to  other 
UAVs’  Situation  Awareness,  reactivity  to  spontaneous  enemy  contacts)  and 
airspace  deconfliction  rules  (minimum  tactical  separation  in  time  and  space).  The 
uncontrollable  factors  to  explore,  which  are  often  ignored  in  other  studies,  include 
the  enemy  Time  Critical  Target’s  parameters  (concealment,  detection  range  and 
vulnerability  duration)  that  characterize  the  shoot-and-scoot  behaviors. 

The  experiment  consists  of  running  the  agent-based  models  over  a  large 
combination  of  design  points  (i.e.,  settings  for  the  controllable  and  uncontrollable 
factors)  and  collecting  the  measures  of  effectiveness.  Regression  and  other 
statistical  analysis  techniques  are  then  used  to  analyze  the  data  and  to  provide 
insights  to  the  following  questions: 

•  What  are  good  cooperative  sensing  network  configurations? 

•  Are  there  diminishing  returns  with  increasing  UAVs  allocated? 

•  What  are  good  UAV  responses  in  a  cooperative  sensing  environment? 

•  What  are  the  effects  of  degraded  communications  and  increasing 
latency? 

•  What  are  the  effects  of  UAV  failure  in  the  presence  of  cooperative 
sensing? 
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Furthermore,  the  analysis  allows  identification  of  main  effects  and 
interactions  which  may  have  significance  impacts  on  ISR  effectiveness  for  a 
cooperative  sensing  network,  thus  providing  areas  of  focus  for  more  detailed 
experiments  and  analysis. 

One  should  be  aware  that  the  employment  of  a  disparate  suite  of  sensors 
is  an  important  consideration  for  cooperative  sensing.  However,  it  is  not  the  focus 
of  this  thesis  to  model  detailed  sensor  performances  against  a  wide  spectrum  of 
terrains  and  targets.  The  sensors  in  the  agent-based  model,  though  they 
consider  factors  of  detection  degradation  imposed  by  the  different  terrain 
features,  are  subjected  to  the  same  degradation  on  each  terrain  feature 
irrespective  of  the  types  of  sensors  and  targets.  The  range  of  sensors  explored  is 
broadly  classified  into  two  main  categories:  the  stand-off  wide  area  sensors  on 
board  the  MAE  and  the  close-in  small  area  sensors  on  board  the  LAE. 

F.  AGENT-BASED  SIMULATION 

Reductions  in  military  operating  budgets  and  improvements  in  computer 

technology  have  driven  the  increasing  use  of  simulations  throughout  the  military. 

Simulations  are  being  employed  in  a  wide  variety  of  military  applications 

including  training,  mission  rehearsal,  system  analysis,  system  acquisition  and 

tactical  decision  aiding.  Furthermore,  high  complexity  and  nonlinearities  in 

combat,  where  outcomes  are  highly  correlated  to  battlefield  conditions  and 

events,  make  it  too  complicated  and  costly  -  if  not  impossible  -  to  explore  a  full 

range  of  possibilities  for  any  useful  study  in  field  experiments.  In  most  of  these 

cases,  simulations  are  used  for  initial  exploration  to  bound  complex  problems 

and  tease  out  areas  of  emphasis  relevant  to  the  questions  at  hand  that  can  then 

be  expounded  more  meaningfully  using  methods  such  as  field  experiments  and 

live  tests  and  evaluations.  The  mission  of  the  Defense  Modeling  and  Simulation 

Office  (DMSO)  is  to  support  the  war-fighter  by  leading  a  defense-wide  team  in 

fostering  interoperability,  reuse,  and  affordability  of  Modeling  and  Simulation 

(M&S),  as  well  as  the  responsive  application  of  these  tools  to  provide 
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revolutionary  war-fighting  capabilities  and  improve  aspects  of  DoD  operations.  As 
a  catalyst  organization  for  the  DoD,  DMSO  ensures  that  M&S  technology 
development  is  consistent  with  other  related  initiatives  by  performing  key 
corporate  level  functions  necessary  to  encourage  cooperation,  synergism,  and 
cost-effectiveness  among  the  M&S  activities  of  the  DoD  Components  (Defense 
Modeling  and  Simulation  Office  website,  2005). 

The  traditional  (and  widely  used)  modeling  technique  in  the  DoD  uses 
differential  equations  to  calculate  casualties  and  changes  in  the  frontlines.  These 
equations  are  called  Lanchester  equations  and  were  originally  published  in  1914. 
Many  modern  simulation  models  use  variants  of  the  Lanchester  equations  to 
predict  the  attrition  rates  of  opposed  forces  massed  in  parallel  strips  across  the 
battlefront.  According  to  a  RAND  study,  “these  models  were  developed  when 
computers  had  much  more  limited  capabilities,  making  it  necessary  to  reduce  the 
number  of  simulation  entities  and  to  use  aggregation  techniques”  (Gonzales,  et 
al. ,  2001).  The  aggregated  Lanchester  equation-based  models  are  beginning  to 
fall  out  of  favor  in  some  DoD  agencies  for  several  reasons.  Tighe  cites  a  paper 
by  Battilega  and  Grange  that  shows  that  the  equations  “do  not  accurately  model 
many  historical  battles”  (Tighe,  1999).  This  is  surprising  given  that  many  of  these 
battles  follow  the  massive  force-on-force  attrition  warfare  formula  for  which  the 
equations  are  designed.  With  modern  warfare  increasingly  reliant  on  C4ISR 
systems,  the  old  models  break  down  even  further.  Gonzales  points  out  that  the 
legacy  models  cannot  model  individual  C4ISR  effects,  and  can  only  be 
represented  by  adjusting  parameters.  He  continues  by  highlighting  the  fact  that 
these  models  cannot  take  into  account  how  information  is  used  to  support 
command  decision-making  processes.  This  severely  limits  their  utility  for 
assessing  many  Information  Superiority  concepts,  such  as  force  synchronization, 
that  may  be  enhanced  or  enabled  by  advanced  C4ISR  systems  (Gonzales,  et  al., 
2001).  In  other  words,  what  good  is  C4ISR  when  your  forces’  movements  are 
scripted? 
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These  problems  have  been  addressed  by  a  new  generation  of  models  that 
attempt  to  model  complex  adaptive  systems.  Forces  are  made  up  of  individual 
“agents”  that  are  programmed  to  follow  a  rough  set  of  rules.  “The  individual 
agents  are  then  responsible  for  making  their  own  decisions  as  to  how  they 
should  prosecute  the  battle”  (Tighe,  1999).  Hence  they  are  adaptive.  While  the 
rules  that  govern  an  individual  agent  may  be  simple,  a  collection  of  agents 
interacting  with  one  another  (and  the  synthetic  environment)  will  exhibit  complex 
behaviors.  These  agent-based  models,  which  are  better  suited  for  the  analysis  of 
modern  concepts  such  as  Network  Enabled  Warfare  and  Effects  Based 
Operations,  have  begun  to  catch  on  in  the  DoD  M&S  community  (Bjorkman  and 
Sheldon,  2002;  North,  et  al.,  2003). 

G.  MAP  AWARE  NON-UNIFORM  AUTOMATA  (MANA) 

The  benefits  of  agent-based  simulation  models  are  greatly  harnessed  by 
the  combat  researchers  and  analysts  in  the  Marine  Corps  Warfighting  Laboratory 
(MCWL).  The  MCWL’s  Project  Albert  is  the  research  and  development  effort 
whose  goal  is  to  develop  the  process  and  capabilities  of  Data  Farming.  Data 
Farming  is  a  method  to  address  decision-maker's  questions  by  applying  high- 
performance  computing  to  relatively  simple  models  in  order  to  examine  and 
understand  the  landscape  of  potential  simulated  outcomes,  enhance  intuition, 
find  surprises  and  outliers,  and  identify  potential  options.  Data  Farming  is  the 
method  by  which  potentially  millions  of  data  points  are  explored  and  captured. 
This  process  is  made  possible,  in  part,  by  the  exploitation  of  High  Performance 
Computing  assets  and  methods,  and  the  project  is  fully  supported  by  the  Maui 
High  Performance  Computing  Center  (MHPCC).  The  Project  Albert  modeling 
approach  is  achieved  through  the  development  of  a  suite  of  models,  sometimes 
called  distillations,  to  drive  home  the  point  that  these  models  are  produced  as  an 
intentional  complement  to  the  very  highly-detailed  simulations  being  developed 
and  used  within  the  DoD,  which  by  the  very  fact  that  they  are  so  highly-detailed 
and  encumbered,  do  not  permit  the  examination  of  a  very  wide  range  of 
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possibilities  and  outcomes.  By  virtue  of  distillations  being  much  easier  to  run  and 
understand,  they  are  proving  to  be  effective  tools  that  help  capture  and 
scientifically  reproduce  the  ideas  of  subject  matter  experts,  such  as  those 
thinking  about  tomorrow's  concepts,  doctrine,  and  requirements.  This  suite  of 
entity-based  models  allow  for  rapid  and  highly  tailorable  changes  in  entity 
characteristics  and  behaviors,  quite  amenable  to,  and  intentionally  designed  for 
rapid,  repeatable  concept  exploration.  Project  Albert  develops  a  suite  of  modeling 
platforms,  rather  than  a  single  realization  of  a  model.  This  has  the  added  benefit 
of  allowing  the  robustness  of  observations  across  modeling  platforms  to  be 
examined.  Also,  each  model  has  inherent  strengths  and  unique  capabilities  with 
regard  to  each  aspect  of  modeling  how  entities  think,  decide,  shoot,  move,  and 
communicate  (Marine  Corps  Warfighting  Laboratory  Project  Albert  website, 
2005). 

The  MANA  software,  developed  by  the  New  Zealand  Defense  Technology 
Agency,  is  one  of  the  distillation  modeling  platforms  extensively  used  by  the 
Project  Albert  Team  to  answer  real  world  questions.  The  authors  of  the  MANA 
User  Manual  (Galligan,  et  al. ,  2004)  present  a  powerful  illustration,  “...the  world 
is  far  more  complicated  than  Newton’s  equations  (laws  of  motion)...  Therefore,  to 
rely  on  models  built  “on  a  bedrock  of  physics”  is  to  deceive  ourselves.  It  is  a  myth 
that  a  more  detailed  model  is  necessarily  a  better  model,  because  it  is  impossible 
to  capture  accurately  every  aspect  of  nature”.  They  further  their  arguments  by 
noting  that  the  non-linear  nature  of  equations  describing  many  real  world 
phenomena  makes  them  extremely  sensitive  to  initial  conditions.  This  means  that 
even  infinitesimal  errors  in  describing  the  real  world  initial  conditions  will  cause 
the  model  to  make  predictions  that  are  almost  uncorrelated  with  actual  events. 
The  illustration  provides  a  clear  motivation  for  the  design  of  MANA  as  what  they 
refer  as  a  scenario-exploring  model. 

MANA  is  in  a  general  class  of  models  called  agent-based  models  (ABMs), 
which  have  the  characteristics  of  containing  entities  that  are  controlled  by  internal 
decision-making  algorithms.  Hence,  an  ABM  combat  model  contains  entities 
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representing  military  units  that  make  their  own  decisions  as  they  react  to  their 
surroundings,  as  opposed  to  the  modeler  explicitly  determining  their  behaviors  in 
advance.  Models  built  in  MANA,  like  those  built  using  other  Project  Albert 
modeling  platforms,  are  sometimes  called  agent-based  distillations  because  their 
common  intention  is  to  model  the  essence  of  a  problem  rather  than  to  describe 
every  aspect  of  a  military  operation.  MANA  is  designed  to  explore  key  concepts 
such  as  Situation  Awareness,  Communications,  Terrain  Features,  Waypoints  and 
Event-Driven  Personality  Changes.  In  MANA,  agents  can  exhibit  a  surprisingly 
wide  set  of  behaviors  as  long  as  careful  configurations  are  given  to  sets  of 
parameters  that  determine  the  agents’  propensities,  move  constraints,  basic 
capabilities  (such  as  sensors,  weapons,  interactions  and  movement  speed)  and 
movement  characteristics  (Galligan,  et  al. ,  2004). 

The  model  is  designed  to  analyze  the  value  of  things  such  as  situation 
awareness,  command  and  control,  and  the  informational  edge  that  enhanced 
sensors  provide.  These  features  are  limited  in  those  models  which  purport  to  be 
detailed,  highly  physics  based  and  rigorous  where  these  aspects  of  combat  can 
only  be  represented  within  the  model  in  a  completely  arbitrary  way  by  the 
modeler  (Galligan,  et  al.,  2004).  These  strengths  of  MANA  as  an  agent-based 
model  ties  in  closely  to  Gonzales’  (2001)  presentation  about  the  limitations  of 
DoD  legacy  models  as  discussed  in  Section  F.  These  are  also  the  strengths  that 
attracted  the  author  to  selecting  MANA  as  a  modeling  tool  for  the  analysis  of 
cooperative  sensing. 

H.  THESIS  ORGANIZATION 

Chapter  II  begins  by  painting  the  modified  Sea  Viking  scenario  which 
includes  the  area  of  operation  and  the  two  sided  force  deployments  and 
objectives.  The  chapter  also  describes  the  model  development  and  in  particular 
the  modeling  parameters  and  techniques  used  to  achieve  the  desired  agent 
behaviors. 


13 


Chapter  III  covers  a  detailed  discussion  of  the  controllable  and 
uncontrollable  factors  of  interest  in  the  analysis  and  their  operational  significance 
and  tradeoffs.  It  also  provides  a  brief  contrast  of  various  possible  designs  of 
experiments  and  concludes  that  a  Nearly  Orthogonal  Latin  Hypercube  design  is 
the  most  efficient  choice  for  running  the  experiments. 

In  Chapter  IV,  the  data  analysis  process  is  presented  and  followed  by  a 
meticulous  exploration  of  the  simulation  results  using  regression  and  statistical 
analysis  techniques. 

Chapter  V  concludes  the  thesis  with  a  presentation  of  operational  insights 
and  recommendations  from  the  results  of  the  analysis.  It  also  presents 
suggestions  for  follow-on  work  that  the  author  considers  worthwhile  exploring. 
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II.  SCENARIO  AND  MODEL  DEVELOPMENT 


A.  THE  SCENARIO 


The  IPB  scenario  prior  to  the  Marine’s  amphibious  assault  depicted  in 
Figure  2  provides  the  basis  for  the  study  of  the  effectiveness  of  a  cooperative 
sensing  network.  The  area  of  operation  extends  across  the  Los  Angeles  and  San 
Diego  city  areas  represented  in  the  map  as  light  orange  regions.  The  Red  target 
types  included  in  this  scenario  represent  two  main  arrays  of  targets  characterized 
by  their  deployment  profiles,  vulnerability  constraints  and  ease  of  detectability. 


Figure  2.  IPB  Scenario  for  Marine  Amphibious  Assault  [Best  Viewed  in  Color] 


Area  5  at  the  upper  right  corner  shows  the  Twenty-nine  Palms 
encompassing  the  Red  objective  and  their  senior  leadership,  and  is  protected  by 
an  armored  battalion.  Area  4  in  the  central  region  defines  the  deployment  ground 
of  the  multiple  launched  rocket  systems  and  the  anti-aircraft  artillery  elements  for 
the  defense  of  the  Red  objective.  These  targets  are  commonly  deployed  in 
isolation  in  a  small  fighting  unit.  They  are  usually  well-concealed  and  have  limited 
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windows  of  vulnerability,  which  makes  detection  and  prosecution  of  these  targets 
very  challenging.  The  three  armor  battalions  concentrated  in  Areas  1,  2  and  3 
move  along  the  axis  as  shown  in  the  map  to  stage  a  barrier  patrol  line  against  the 
coastal  assault.  The  armor  concentrations  represent  target  arrays  that  are 
commonly  deployed  in  formations  and  are  “noisy”  from  the  electronic  and  visual 
detection  perspectives,  hence  making  them  easier  targets  for  detection  and 
prosecution.  Area  6  represents  the  Blue  MAE  and  LAEs  supporting  the  IPB 
operation  for  the  MEB.  Each  LAE  has  a  pre-assigned  area  of  responsibility  to 
provide  classification  of  enemy  contacts,  while  the  single  MAE  sweeps  through 
the  whole  area  of  operation  to  provide  wide  area  sensor  coverage. 


B.  MODEL  DEVELOPMENT 

1.  Environment 

The  terrain  shown  in  the  Figure  3  is  created  by  Raffetto  (2004).  The  dark 
yellow  in  the  terrain  depiction  represents  city  areas;  the  blue  regions  signify 
bodies  of  water;  the  brown,  dark  and  light  green  regions  identify  the  desert, 
dense  and  sparse  forested  areas  respectively;  the  light  yellow  lines  show  the 
highways  and  major  road  networks  in  the  cities.  The  terrain  features  are 
associated  with  parameters  that  affect  the  movement  speed,  engagement 
effectiveness,  and  concealment  of  the  agents.  These  values  provide  multipliers 
to  compute  the  overall  effective  speed,  kill  and  detection  outcomes  of  the  agents 
at  every  time  step.  The  blue  aircraft  icons  depict  the  Blue  MAE  and  LAEs;  the  red 
icons  represent  the  Red  armored  vehicles  and  time  critical  targets;  the  pink  icons 
identify  the  neutral  elements. 
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Figure  3.  MANA  Model  for  IPB  Scenario  [Best  Viewed  in  Color] 


The  battle-space  extends  across  32.6  to  35.1  degrees  in  latitude  and  - 
1 18.8  to  -1 15.6  degrees  in  longitude,  which  equates  to  160  nm  in  width  and  150 
nm  in  height.  The  maximum  spatial  resolution  that  can  be  represented  in  MANA 
is  1000  by  1000  cells,  which  computes  to  an  average  of  approximately  287  by 
287  meters  in  both  length  and  width  per  cell  for  this  scenario.  The  model  is  set  to 
elapse  10  real-time  seconds  for  every  time  step  taken  for  the  simulation.  Since 
the  total  amount  of  time  for  the  IPB  operations  takes  four  hours,  the  simulation 
terminates  after  1440  time  steps  have  elapsed.  The  conversions  between  MANA 
units  and  real-world  units  are  tabulated  in  a  spreadsheet  as  illustrated  in  Figure  4 
to  facilitate  subsequent  unit  conversions  in  the  development  of  the  models. 
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2.  Time  Critical  Target 

Time  Critical  Targets  (TCTs)  are  targets  requiring  immediate  response 
because  they  either  pose  (or  will  soon  pose)  a  danger  to  friendly  forces,  or  they 
are  highly  lucrative,  fleeting  targets  of  opportunity  with  an  extremely  limited  time 
window  of  vulnerability,  the  attack  of  which  is  critical  to  ensure  the  successful 
execution  of  the  subsequent  battle  operations.  The  Red  MLRS  and  ADA 
elements  in  the  model,  totaled  30,  are  classified  as  time  critical  targets  as  they  a 
limited  windows  of  vulnerability  characterized  by  their  shoot-and-scoot  tactics. 
These  targets  are  generally  deployed  in  terrains  where  concealment  to  escape 
enemy’s  detection  is  almost  perfect.  The  TCT  comes  out  of  its  “hiding  place”  to  a 
nearby  operational  spot  only  when  it  is  ready  to  fire  a  salvo  or  there  is  a  need  to 
reload  its  ammunitions,  and  then  goes  right  back  into  “hiding”.  The  TCT  only 
subjects  itself  to  detection  and  engagement  vulnerabilities  during  this  limited 
period  of  vulnerability  which  generally  last  no  longer  than  30  minutes,  hence 

acquiring  and  prosecuting  such  targets  is  challenging. 
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The  TCT  in  the  model  is  developed  by  creating  a  cycle  of  waypoints  and 
altering  the  behaviors  of  the  agents  in  each  state.  The  agent  moves  from  the 
current  waypoint  to  the  next  during  the  RunStart  state  and  stays  in  this  state  for  a 
defined  vulnerable  duration  which  will  eventually  be  data  farmed.  Once  it  reaches 
the  next  waypoint  and  the  vulnerability  duration  elapses,  the  agents  change  into 
the  ReachFinalWaypoint  state  where  the  concealment  rate  (another  parameter  to 
be  data  farmed)  is  applied  to  these  targets  to  emulate  the  “hide”  behaviors.  The 
TCT  is  assumed  to  have  the  capability  to  detect  an  approaching  UAV  with  its 
counter-detection  sensor  and  evade  detection  by  the  UAV  by  setting  the 
concealment  rate  in  the  EnemyContact  state.  The  TCT  loops  through  the  cycle 
until  it  is  detected  and  classified  by  a  Blue  UAV  as  shown  in  the  state  transition 
diagram  of  Figure  5. 


Figure  5.  State  Diagram  of  a  Time  Critical  Target  Agent 


3.  Armor  Battalion 

The  model  consists  of  four  armor  battalions;  each  is  implemented  as  a 
separate  squad  and  comprises  40  medium  tanks.  The  three  coastal  battalions 
are  scripted  to  follow  pre-defined  routes  along  the  road  axes  and  the  last 
battalion  is  setup  to  patrol  around  the  parameter  of  Twenty-nine  Palms.  The 
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agents  are  set  up  with  propensities  (or  weights,  as  commonly  known  in  many 
agent-based  models)  to  move  towards  the  next  waypoint  and  to  move  on  terrains 
that  are  easy-going,  in  this  case,  to  follow  the  road  networks  whenever  possible. 


4.  MAE  and  LAE  UAVs 

Due  to  the  considerably  limited  area  of  operation,  it  is  operationally 
infeasible  to  devote  more  than  one  MAE  like  the  Predator  to  support  the  mission. 
However,  an  amphibious  strike  force  commander  may  have  up  to  five  LAEs  like 
the  Pioneer  at  his  disposal.  While  the  MAE  provides  wide  area  surveillance  and 
reconnaissance  for  the  entire  area  of  operation  with  poor  classification  capability, 
an  LAE  can  provide  effective  close-in  classification  of  the  enemy  contacts  within 
its  higher  resolution  sensor  range.  The  models  characterize  these  UAV 
parameters  by  varying  the  detection  and  classification  ranges  and  the 
corresponding  probabilities. 

The  UAVs  are  given  a  “weapon”  to  shoot  at  the  targets  with  range  and 
probability  equal  to  the  range  and  probability  for  sensor  classification, 
respectively.  This  means  that  a  target  is  “killed”  when  being  classified  to  prevent 
repeated  classification  of  the  same  target.  The  “weapon”  is  loaded  with  a  very 
large  amount  of  “ammunition”  to  ensure  the  ammunition  is  not  depleted. 

To  support  the  analysis  of  the  effects  of  LAE  failure  on  ISR  effectiveness 
of  the  cooperative  sensing  network,  one  UAV  killer  agent  is  created  to  “shoot” 
and  disable  a  single  LAE  after  a  pre-determined  time  from  the  beginning  of  the 
scenario.  The  UAV  killer  is  randomly  positioned  in  the  area  of  operation  and 
remains  inactive  in  the  Default  state  until  it  is  time  for  an  LAE  to  fail.  It  then 
switches  to  another  state  where  the  “weapon”  is  enabled  and  “fires”  a  single  shot 
with  a  probability  of  kill  equal  to  1 .0  at  the  nearest  LAE,  hence  disabling  that  LAE. 
The  MAE  in  the  model  is  assumed  to  be  100%  reliable  over  the  four-hour  IPB 
operation,  as  MAEs  are  generally  designed  for  longer  endurance  missions  and 
with  longer  mean-time-to-failure  parameters. 
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The  movements  and  interactions  of  the  UAV  are  characterized  by  a  set  of 
propensities  in  the  model.  Each  UAV’s  responsiveness  to  either  an  incoming 
contact  from  another  UAV  or  a  newly  sighted  enemy  contact  determines  its 
tendency  to  be  diverted  from  its  original  route.  The  propensity  for  each  UAV  to 
move  towards  the  next  waypoint  is  set  to  50.  By  changing  the  propensities  to 
move  towards  enemy  in  the  Agent  SA  properties,  and  move  towards  unknown  in 
the  Inorganic  SA,  the  reactivity  of  the  UAV  to  such  targeting  information  can  be 
explored.  Additionally,  the  airspace  deconfliction  parameter  which  determines  the 
spatial  separation  of  the  UAVs  in  flight  is  varied  by  changing  the  propensities  to 
move  away  from  friends  in  the  models.  Further  details  on  how  these  propensities 
are  data  farmed  will  be  discussed  in  Chapter  III. 

It  should  also  be  noted  that  the  effectiveness  of  enemy  contact  detection 
and  classification  are  very  sensitive  to  the  routes  taken  by  the  UAVs  due  to  the 
relatively  small  UAV  sensor  coverage  compared  to  the  area  of  operation.  If  no 
UAV  flies  near  a  target,  there  will  be  neither  a  detection  nor  a  classification.  The 
routes  for  the  LAEs  are  developed  based  on  intuitive  justifications  after  analyzing 
the  terrain  and  target  profiles.  Refer  to  Appendix  A  for  the  design  rationale  and 
snapshots  of  the  set  of  MAE  and  LAE  routes.  For  the  development  of  these 
routes,  the  author  consulted  with  Captain  Starr  King,  USN,  NWDC-Sponsored 
Chair  of  Warfare  Innovation  and  Chair  of  Applied  Systems  Analysis  of  the  Naval 
Postgraduate  School  and  Captain  Kevin  McMindes,  a  USMC  pilot;  both  consider 
them  reasonable  and  sufficiently  detailed  for  the  purpose  of  modeling  and 
analysis  to  answer  the  questions  of  interest.  However,  one  should  not  preclude 
other  alternative  routes  based  on  various  operational  judgments  and  objectives. 

5.  UAV  Communications  Network 

A  communications  network  is  set  up  in  MANA  to  facilitate  the  sharing  of 

information  between  the  MAE  and  the  LAEs,  as  illustrated  in  Figure  6.  Two  types 

of  situational  awareness  maps  are  provided  in  MANA:  a  squad  map  which  holds 

direct  squad  contact  memory,  and  an  inorganic  map  which  stores  contact 
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memories  provided  by  other  squads  through  communications  links  (Galligan,  et 
al. ,  2004).  The  contacts  discovered  by  the  MAE  are  sent  through  the 
communications  links  and  updated  on  the  inorganic  map  of  the  LAE  agents,  and 
vice  versa.  The  data  links  for  both  MAE  and  LAE  are  C-Band/UHF  and  are 
assumed  to  have  communications  ranges  sufficient  for  coverage  throughout  the 
entire  area  of  operation  as  long  as  there  is  Line  Of  Sight  (LOS)  between  the 
UAVs.  Although  modern  communications  technologies  offer  superb  link  quality 
and  availability  from  a  technical  view,  communications  degradation  is,  none  the 
less,  an  important  consideration  in  most  military  networks  in  which  the  systems 
are  deployed  due  to  the  adverse  operating  environments  and  weather  conditions. 
Hence,  factors  related  to  communications  degradation  are  considered  in  the 
models  by  changing  the  values  corresponding  to  message  delay  and  link 
reliability  over  each  communications  link. 

MAE 


Figure  6.  Information  Flow  Between  the  MAE  and  the  LAEs  in  the  UAV  Network  for 

Shared  Situation  Awareness 


In  actual  operational  environments,  message  latency  represents  the  delay 
from  the  time  a  message  is  sent  from  the  source  UAV  to  the  time  it  is  received  at 
the  destination  UAV,  and  varies  depending  on  the  distance  between  the  two 
nodes  and  the  current  loading  on  the  network.  The  link  reliability  in  the  model 
emulates  the  availability  of  the  communications  links  for  transmissions.  There 
could  be  times  where  the  LOS  between  two  UAVs  for  transmission  is  not 
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available,  in  particular,  when  they  are  flying  at  low  altitudes  across  rough  terrains. 
Another  major  threat  to  link  reliability  could  be  the  enemy’s  act  of  electronic 
attack,  which  is  a  prevalent  form  of  interdiction  in  modern  warfare. 

C.  MEASURES  OF  EFFECTIVENESS 

The  DMSO  defines  a  Measure  of  Effectiveness  (MOE)  as  “a  qualitative  or 
quantitative  measure  of  the  performance  of  a  model  or  simulation  or  a 
characteristic  that  indicates  the  degree  to  which  it  performs  the  task  or  meets  an 
operational  objective  or  requirement  under  specified  conditions”  (Defense 
Modeling  and  Simulation  Office  Online  M&S  Glossary,  2005).  This  study 
measures  the  expected  proportion  of  Red  TOT  and  armor  entities  classified  at 
the  end  of  the  four-hour  IPB  to  assess  the  effectiveness  of  employing 
cooperative  sensing  to  detect  each  type  of  targets.  The  MOEs  distinguish 
between  different  target  types;  this  could  potentially  provide  insights  to  the 
decision  makers  about  appropriate  ways  to  apply  different  UAV  tactics  and 
behaviors  depending  on  the  target  characteristics  and  their  deployment  profiles 
in  the  presence  of  a  cooperative  sensing  environment.  For  instance,  when  an 
LAE  establishes  a  new  armor  contact,  it  may  be  inclined  to  follow  the  contact 
knowing  that  armors  tend  to  operate  in  formation.  This  could  result  in  a  higher 
classification  rate.  In  contrast,  following  a  TCT  contact  may  not  result  in  a  higher 
classification  rate  since  TCTs  tend  to  deploy  in  isolation. 
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III.  DESIGN  OF  EXPERIMENT 


A.  EXPERIMENTAL  FACTORS 

1.  Controllable  Factors  -  Blue  ISR  Parameters 

The  number  of  factors  affecting  a  cooperative  sensing  network’s 
effectiveness  is  large,  and  each  factor  may  have  a  wide  variation  of  potential 
levels  (i.e.,  settings).  The  factors  are  often  confounded  and  some  even  exhibit 
conflicting  interactions  among  them.  For  instance,  it  may  appear  obvious  that  a 
faster  UAV  is  desirable  because  it  can  cover  a  larger  sweep  area  in  a  fixed 
duration  of  time.  However,  in  most  sensor  systems  involving  human  operators,  a 
minimum  integration  time  is  needed  for  a  target  to  “remain  visible”  within  the 
sensor  field  of  view  to  be  considered  a  positive  detection.  As  the  UAV  travels 
faster,  the  probability  of  detection  and  classification  decreases  since  the  time 
available  for  the  sensor  to  look  at  the  target  reduces.  From  an  endurance 
standpoint,  a  faster  UAV  is  generally  less  fuel  efficient,  thus  has  less  endurance 
than  a  slower  UAV.  Another  case  for  conflicting  interactions  between  factors 
includes  the  optimal  use  of  MAE  and  LAE  UAVs.  The  MAEs,  in  general,  provide 
wider  sensor  coverage  than  the  LAEs  due  to  their  higher  operational  altitudes.  In 
reality,  higher  altitudes  are  associated  with  reduced  signal  strengths,  reduced 
angular  resolutions,  and  increased  interferences;  all  of  these  are  culprits  for 
driving  down  the  probabilities  of  detection  and  classification.  Similarly,  it  may 
appear  desirable  to  have  a  wider  Field  Of  View  (FOV)  in  compensation  for  lower 
altitude  flights  to  have  an  equivalent  sweep  area.  Again,  one  would  quickly  end 
up  sliding  on  the  tradeoff  curve  between  a  wide  FOV  with  low  resolution  and 
narrow  FOV  with  high  resolution,  and  imagery  resolutions  have  direct  impact  on 
the  target  classification  abilities.  These  illustrations  clearly  present  the  case  that 
many  of  these  factors  are  closely  interrelated;  the  analysis  of  their  interactions  is 
not  trivial. 
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Factors  are  classified  as  controllable  in  the  simulation  experiments  if  they 
represent  action  options  to  decision  makers  for  the  real-world  problems.  For  this 
reason,  controllable  factors  are  also  commonly  known  as  decision  factors.  In 
concert  with  cooperative  sensing,  the  decision  factors  that  potentially  contribute 
to  the  effectiveness  of  the  sensing  network  in  performing  ISR-related  missions, 
thus  having  direct  relevance  to  gaining  insights  in  our  analysis,  include: 

Factors  related  to  sensing  network  configurations 

•  Number  of  LAEs  to  one  MAE 

Factors  related  to  UAV  parameters 

•  Airspeed 

•  Detection  range 

•  Classification  range 

•  Classification  probability 

Factors  related  to  network  parameters 

•  Link  reliability 

•  Message  latency 

Factors  related  to  UAV  routing  behaviors 

•  Reactivity  to  friendly  UAV  SA  versus  tendency  to  follow  a  planned 
route 

•  Reactivity  to  a  newly  sighted  enemy  contact  versus  tendency  to  follow 
a  planned  route 

Factors  related  to  airspace  deconfliction 

•  Minimum  separation  between  UAVs 
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The  UAVs  selected  for  the  study  are  not  intended  to  be  homogeneous  but 
broadly  classified  into  two  categories,  namely  the  MAE  and  LAE.  The 
simultaneous  investigation  over  the  two  categories  of  UAVs  allows  one  to  identify 
factors  which  are  deemed  important  to  the  questions  to  be  expounded,  therefore 
provides  further  design  considerations  for  architecting  a  cooperative  sensing 
network  for  operational  deployment.  The  factors  include  UAV  airspeed,  ranges 
for  detection  and  classification,  and  detection  and  classification  probabilities.  To 
ensure  the  model  is  valid,  the  factors  are  varied  with  ballpark  values  based  on 
current  to  near-future  published  technologies.  These  materials  can  be  easily 
found  from  open-source  literatures  and  databases  such  as  The  JANES  Defense 
databases,  The  UAV  Forum  and  Federation  of  American  Scientist  websites.  The 
MAEs  in  the  model  have  operational  airspeed  ranging  from  60  to  100  knots  and 
sensor  detection  range  from  5,000  to  12,000  meters.  The  LAEs  have  operational 
airspeed  ranging  from  40  to  65  knots  and  sensor  detection  range  from  2,000  to 
7,000  meters.  In  the  model,  the  classification  range  is  set  no  farther  than  the 
detection  range  by  taking  values  from  0.6  to  1 .0  of  the  detection  range. 

It  should  be  obvious  that  the  ability  of  a  UAV  to  detect  and  classify  ground 
targets  is  dependent  on  its  sensor  coverage.  The  sensor  coverage  for  a  specified 
FOV  increases  with  the  UAV  flight  altitude  as  shown  in  Figure  7.  The  MAEs  in 
the  model  have  an  operational  altitudes  ranging  from  20,000  to  35,000  ft  above 
ground  level  (AGL)  while  the  LAEs  from  5,000  to  20,000  ft  AGL.  The  explored 
range  of  sensor  FOV  stretches  from  30°  to  60°.  Figure  7  displays  the  resulting 
range  of  possible  sensor  coverage  considered  in  the  study.  As  an  illustration,  a 
tactical  LAE  flying  at  15,000  ft  AGL  carrying  an  EO  payload  with  a  60°  FOV  has 
circular  sensor  coverage  with  radius  2,640  meters  (i.e.,  8,660  ft).  If,  on  detecting 
an  unknown  target,  the  same  sensor  decides  to  close  in  and  lower  its  altitude  to 
10,000  ft  AGL  and  switch  the  EO  camera  to  a  30°  FOV  in  order  to  classify  the 
target,  it  then  has  a  sensor  coverage  radius  of  817  meters. 
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UAV  Sensor  Coverage  vs  Altitude  Chart 


Figure  7.  Graph  of  UAV  Detection  Range  As  a  Function  of  Altitude  and  Field  of 

View  [Best  Viewed  in  Color] 


A  complex  and  detailed  representation  that  mimics  a  real  world  sensor  in 
the  model  is  viewed  unnecessary,  in  the  author’s  opinion,  for  this  analysis.  It  is 
not  the  intent  of  the  study  to  focus  on  highly-detailed  and  cumbersome  physical- 
based  modeling  to  give  point  estimate  results.  Instead,  a  simple  model  facilitates 
rapid  execution  to  tap  the  power  of  agent-based  distillations  to  provide  insights 
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on  the  outcomes  for  varying  model  inputs.  In  this  regard,  cookie  cutter  models 
are  assumed  sufficient  for  detecting  or  classifying  targets;  targets  falling  within 
the  sensor  coverage  have  equal  chance  of  being  detected  or  classified.  The  LAE 
classification  probability  has  values  from  0.3  to  1.0.  The  MAE  classification 
probability  takes  on  values  from  0.1  to  0.8  of  the  LAE  classification  probability  to 
ensure  that  the  latter  is  always  lower  than  the  former  for  every  simulation  run. 

One  of  the  objectives  as  part  of  the  study  involves  finding  good  sensing 
network  configurations  to  support  swift  maneuver  and  precision  engagement  by 
the  MEB.  In  the  model,  a  single  MAE  having  large  sensor  coverage  for  effective 
initial  detection  is  supplemented  by  multiple  LAEs  to  enable  effective  close-in 
target  classification.  The  number  of  LAEs  employed  affects  how  much  area  can 
be  covered  and  how  long  it  takes  to  complete  the  search.  An  upper  limit  of  five 
LAEs  has  been  selected  for  the  study.  Although  Raffetto  suggests  that  it  is 
unlikely  that  the  Marine  Corps  will  have  the  resources  and  personnel  to  employ 
more  than  three  UAVs  simultaneously  as  a  current  standard  operating  procedure 
for  a  MEB  size  area  of  operation  (Raffetto,  2004),  the  addition  of  a  fourth  and  fifth 
LAE  is  envisioned  for  future  ISR  capability  extension  due  to  constant  reduction  in 
the  cost  of  acquiring  and  manning  UAVs.  The  additional  LAEs  allow  evaluation  of 
overall  ISR  effectiveness  as  the  number  of  LAEs  increases  in  a  cooperative 
sensing  environment. 

The  communications  network  forms  the  essential  backbone  for  sharing 
detection  information  between  the  MAE  and  the  LAEs.  Degraded 
communications  will  inevitably  shortchange  the  full  potential  one  can  harness 
from  a  cooperative  sensing  network  to  achieve  high  ISR  capability.  As  discussed 
in  the  preceding  chapter,  electronic  attack  is  becoming  more  prevalent  in  modern 
warfare.  Coupled  with  the  adverse  weather  and  environmental  conditions  that 
these  UAV  networks  are  likely  to  operate  in,  the  transmission  reliability  and 
accuracy  of  such  information  over  the  network  must  be  considered.  Link  reliability 
and  message  delay  are  considered  in  the  model  to  emulate  such  degradation  in 
the  communications  network.  While  link  reliability  takes  on  values  from  0%  to 
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100%,  the  latency  at  which  messages  are  transmitted  from  one  UAV  to  another 
varies  from  0  to  10  time  steps  in  MANA,  which  corresponds  to  0  to  100  seconds 
in  real  time. 

The  interactions  and  behaviors  amongst  UAVs  within  the  network  are 
expected  to  have  significant  impacts  on  the  effectiveness  of  a  cooperative 
sensing  network.  The  responsiveness  of  a  UAV  to  other  UAV  SA  determines  how 
much  the  UAV’s  next  course  of  action  is  influenced  by  incoming  targeting 
information  from  the  network  with  respect  to  its  original  course  of  action.  In 
MANA,  this  is  achieved  by  changing  the  movement  propensities  of  the  UAV 
agents  in  response  to  an  unknown  contact  in  the  inorganic  SA.  The  ranges  of 
weightings  for  these  propensities  vary  from  0  to  100,  and  the  responsiveness  to 
the  other  UAV  SA  is  computed  as: 

(Move  toward  unknown  -  Follow  waypoint)  /  (Follow  waypoint) 

In  the  model,  the  follow  waypoint  weight  is  fixed  at  a  value  of  50;  hence  the 
responsiveness  to  other  UAV  SA  ratio  amounts  to  a  range  of  -1.0  (completely 
unresponsive)  to  +1 .0  (completely  responsive). 

The  other  parameter  that  affects  the  behavior  of  a  UAV  is  its  reactivity  to  a 
newly  sighted  enemy  contact.  In  reality,  this  defines  the  ISR  tactics  that  permit 
dynamic  redirection  of  UAVs  from  their  intended  routes  toward  newly  detected 
hostile  targets.  In  MANA,  this  is  implemented  by  varying  the  movement 
propensity  of  the  UAV  agent  towards  unknown  targets  with  respect  to  its 
propensity  to  follow  waypoints.  As  in  the  previous  metric,  the  response  to  a  newly 
sighted  target  is  computed  as: 

(Move  toward  enemy  -  Follow  waypoint)  /  (Follow  waypoint) 

The  weight  for  moving  toward  an  enemy  ranges  from  0  to  100,  giving  the 
responsiveness  to  a  newly  sighted  enemy  contact  ratio  of  -1.0  (completely 
unresponsive)  to  +1 .0  (completely  responsive). 
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The  UAVs  in  the  scenario  are  separated  in  time  and  space.  This  models 
the  tactical  airspace  control  rules  that  necessitate  UAV  controllers  to  plan  for 
airspace  deconfliction  in  any  UAV  operations.  The  analysis  investigates  the 
effects  far  and  close  airspace  separations  have  on  the  effectiveness  of 
cooperative  sensing  by  modifying  the  UAV  agent’s  propensity  to  move  toward  or 
away  from  uninjured  friends  in  MANA.  A  value  of  0  corresponds  to  closer 
allowable  airspace  separation,  while  -100  leads  to  farther  separation.  These 
values  represent  the  agents’  propensities  to  move  toward  or  away  from  one 
another,  and  not  the  actual  minimum  separation  distances  between  them. 


2.  Uncontrollable  Factors  -  Red  Target  Parameters 

Just  as  controllable  factors  are  important  in  gaining  insights  about  how 
they  impact  the  ISR  effectiveness  of  a  cooperative  sensing  network, 
uncontrollable  factors,  which  are  often  overlooked  by  operational  analysts  and 
decision  makers,  are  of  equal  importance  in  this  analysis,  particularly  since 
simulation  is  employed.  In  a  mathematical  modeling  activity  such  as  simulation 
one  does,  after  all,  get  to  control  everything  whether  or  not  it  is  actually 
controllable  in  the  real  world.  By  exploring  over  ranges  of  settings  for  the 
uncontrollable  factors,  also  known  as  noise  factors,  one  could  benefit  by 
observing  the  influences  these  factors  have  on  the  system  outcomes.  These 
insights  make  the  analyst  more  capable  of  deriving  settings  for  the  controllable 
factors  that  make  the  system  more  robust  to  a  wider  background  variation, 
instead  of  one  that  optimizes  decision  factors  after  fixing  each  noise  factor  to  a 
single  value  which  is  likely  to  deviate  from  reality.  The  robust  design  approach 
was  first  advocated  by  Taguchi  for  quality  planning  and  engineering  product 
design  activities,  and  focuses  on  minimizing  variation  and  sensitivity  to  noise 
(Taguchi  and  Wu,  1980).  Sanchez  (2000)  suggests  the  following  benefits  of 
using  a  robust  design  approach  for  simulation: 

•  Fewer  surprises  when  decision  is  implemented 
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•  Improved  communication  between  analyst  and  “client”  via  expected 
loss 

•  Ability  to  evaluate  trade-offs  between  noise  reduction  costs  and 
performance  quality 

•  Facilitates  continuous  improvement  by  seeking  to  reduce  variability  of 
response,  not  just  achieving  targeted  value,  and 

•  Insights  gained  allow  simultaneous  improvement  in  performance  and 
reduction  in  costs 

In  the  case  of  this  analysis,  valuable  insights  pertaining  to  the  robustness 
of  the  cooperative  sensing  network  can  be  achieved  by  varying  those  factors 
affecting  the  behaviors  of  the  enemy’s  forces.  In  particular,  the  TCT’s  shoot-and- 
scoot  tactics  are  perceived  to  have  significant  impact  on  the  UAVs’  detection  and 
classification  capability.  The  set  of  target  parameters  that  contributes  to  its 
susceptibility  to  UAV  detection  include: 

•  Concealment  factor 

•  Vulnerability  time  window 

•  Sensor  range  for  detecting  nearby  UAVs 

The  target  concealment  factor  has  a  range  of  0%  to  100%.  The  latter 
signifies  that  the  target  is  completely  invisible  to  passing  UAVs;  the  former  adds 
no  additional  reduction  factor  to  the  detection  probability  other  than  those  already 
imposed  by  the  terrain.  The  vulnerability  time  window  describes  the  time  duration 
for  which  a  TCT  exposes  itself  as  it  moves  from  its  hiding  position  to  its  firing 
position  or  next  deployment,  thereby  being  vulnerable  to  UAV  detections.  The 
vulnerability  duration  varies  from  10  to  30  real  time  minutes  or  60  to  180  MANA 
time  steps.  The  TCTs  are  assumed  to  have  the  capabilities  to  detect  incoming 
UAVs  and  take  positions  for  concealment  thus  reducing  the  signatures 
susceptible  to  detection.  Whether  the  target  can  successfully  take  cover  depends 
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largely  on  its  sensor  range  for  detecting  nearby  UAVs  to  provide  sufficient 
response  time  for  concealment.  The  TCT  onboard  sensor  range  varies  from  0  to 
10  kilometers. 

One’s  intuition  suggests  the  reliability  of  UAVs  may  significantly  impact  the 
robustness  of  a  cooperative  sensing  network.  In  addition  to  the  base  case,  the 
overall  classification  effectiveness  is  explored  for  situations  when  a  UAV  fails  or 
crashes,  thus  becoming  unavailable  in  the  IPB  process.  As  the  MAE  in  the 
sensing  network  is  a  critical  node  connecting  and  coordinating  the  ISR  efforts 
amongst  the  LAEs,  it  is  assumed  in  this  study  that  the  MAE  has  a  long  mean 
time  to  failure  and  the  likelihood  of  its  failure  during  the  four  hour  IPB  operation  is 
negligible.  Instead,  the  study  expounds  on  the  degree  the  ISR  capability  is 
degraded  should  a  single  LAE  fail  after  some  time  into  the  operation,  and 
evaluates  how  cooperative  sensing  could  mitigate  the  “hole”  in  ISR  plan.  The 
model  emulates  a  random  failure  to  one  of  the  LAEs  at  the  zero,  first,  second, 
third  and  fourth  hour.  It  should  be  noted  that  an  LAE  failure  at  the  beginning  of 
the  IPB  is  nearly  equivalent  to  the  configuration  of  having  one  less  LAE  from  the 
start,  but  will  have  a  different  and  varied  set  of  UAV  routings  since  the  failed  LAE 
is  chosen  at  random.  Also,  an  LAE  failing  at  the  fourth  hour  has  no  effect  on  the 
classification  outcome  for  the  IPB  since  it  fails  right  at  the  end  of  the  scenario. 
This  is  a  computationally  efficient  way  to  obtain  some  insights  about  the  effects 
of  LAE  reliability  without  having  to  go  into  the  details  of  modeling  UAV 
breakdowns. 

The  ranges  of  factors  in  both  real  world  and  MANA  representations  are 
depicted  in  Figure  8. 
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REAL  WORLD  &  MANA  METRICS 


Factors 

MANA  Names 

Units 

Actual 

MANA 

Low 

High 

Low 

II  High 

Sensing  Network  Config 

#  LAE 

#  LAE 

- 

1 

5 

1 

5 

UAV  Parameters 

MAE  airspeed 

MAE  Airspeed 

knot 

60 

100 

108 

180 

LAE  airspeed 

LAE  Airspeed 

knot 

40 

65 

72 

117 

MAE  detection  range 

MAE  Detect  Range 

km 

5 

12 

17 

42 

LAE  detection  range 

LAE  Detect  Range 

km 

2 

7 

7 

24 

MAE  classification  range 

MAE  Class  Range 

km 

3 

12 

10 

42 

LAE  classification  range 

LAE  Class  Range 

km 

1.8 

7 

6 

24 

MAE  classification  probability 

MAE  Class  Prob 

- 

0.1 

0.8 

0.1 

0.8 

LAE  classification  probability 

LAE  Class  Prob 

- 

0.3 

1.0 

0.3 

1.0 

Network  Parameters 

Link  reliability 

Comms  Reliability 

percent 

0 

100 

0 

100 

Message  latency 

Comms  Latency 

second 

0 

100 

0 

10 

UAV  Routing  Behaviors 

Reactivity  to  friendly  UAV  SA 
versus  tendency  to  follow  a 
planned  route 

Reactivity  to  a  newly  sighted 
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Figure  8.  Experimental  Factors  and  Levels  in  Both  Real  World  Metrics  and  MANA 

Representations 


B.  ORTHOGONAL  LATIN  HYPERCUBE  DESIGN 

In  simulation,  experimental  design  provides  a  way  of  deciding  before  the 
runs  are  made  which  particular  configurations  to  simulate  so  that  the  desired 
information  can  be  obtained  with  the  least  amount  of  simulating.  Carefully 
designed  experiments  are  much  more  efficient  than  a  “hit-or-miss”  sequence  of 
runs  in  which  one  simply  tries  a  number  of  alternative  configurations 
unsystematically  to  see  what  happens.  Factorial  and  fractional  factorial 
experimental  designs  are  particularly  useful  in  the  early  stages  of 
experimentation,  when  one  is  pretty  much  in  the  dark  about  which  factors 
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(parameters  under  investigation  which  are  deemed  to  contribute  to  the  effects  of 
an  experimental  outcome)  are  important  and  how  they  might  affect  the 
responses.  As  the  model’s  behavior  is  gradually  understood  and  a  set  of  more 
specific  goals  are  determined,  a  whole  variety  of  specific  experimental 
techniques  can  then  be  used  to  seek  optimal  combinations  of  factor  levels  that 
maximize  understanding  toward  these  goals  (Law  and  Kelton,  2000). 

The  full  factorial  experimental  design  quickly  becomes  unmanageable 
because  the  number  of  simulation  runs  needed  escalates  exponentially  with 
increasing  number  of  factors  and  factor  levels.  In  this  exploration  of  the  use  of 
multiple  cooperative  UAVs  for  ISR  missions,  the  total  number  of  factors  in 
consideration  is  19  with  some  factors  having  as  many  as  10  levels.  It  works  out 
that  1019  (10  quintillion)  design  points  are  needed  to  study  all  possible  factor-level 
combinations.  If  it  is  desirable  to  make  10  replications  per  design  point,  certainly 
a  modest  sample  size  from  a  statistical  viewpoint,  the  total  number  of  simulation 
runs  would  sum  up  to  102°  (100  quintillion).  If  every  replication  takes  a  computing 
cluster  one  second  to  complete,  the  simulation  would  not  finish  even  when  the 
sun  goes  out  and  the  earth  dies;  a  problem  one  would  not  even  think  of  solving. 

Fractional  factorial  designs  provide  a  way  to  get  good  estimates  of  only 
the  main  effects  and  perhaps  two-way  interactions  at  a  fraction  of  the 
computational  effort  required  by  a  full  factorial  design.  Basically,  a  2k'p  fractional 
factorial  design  is  constructed  by  choosing  1/2P  of  all  the  possible  design  points 
and  then  running  simulations  for  only  these  chosen  points.  Clearly,  one  would 
like  p  to  be  large  from  a  computational  efficiency  viewpoint,  but  a  larger  p  may 
also  result  in  less  information  from  the  experiment,  as  one  might  suspect. 
Furthermore,  the  2k'p  fractional  factorial  experimental  design  does  not  allow  the 
investigation  of  non-linearity  effects. 

A  smarter  design  of  experiment  is  desired.  The  Orthogonal  Latin 

Hypercube  (OLH)  design  was  pursued  for  its  efficiency,  excellent  space  filling 

properties  and  design  flexibility.  Excellent  space  filling  results  in  low  correlation 

between  input  factors;  design  flexibility  imposes  few  restrictions  on  the  number  of 
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factors  and  levels,  and  thus  provides  the  ability  to  identify  nonlinear  relationships. 
The  Nearly  Orthogonal  Latin  Hypercube  (NOLH)  has  nearly  the  same  properties 
but  a  small  amount  of  orthogonality  is  allowed  in  order  to  get  better  space-filling. 
A  good  space-filling  design  is  one  in  which  the  design  points  are  scattered 
throughout  the  experimental  region  with  minimal  unsampled  regions;  that  is,  the 
voided  regions  are  relatively  small.  This  means  that  the  design  points  are  not 
concentrated  in  clusters  or  solely  at  corner  points  of  the  region,  as  can  happen 
with  two-level  factorial  designs  (Cioppa,  2002;  Cioppa  and  Lucas,  2006).  For  a 
general  discussion  of  designing  simulation  experiments,  see  Kleijnen  et  al. 
(2005). 

To  construct  the  NOLH  for  this  analysis,  we  used  a  Microsoft  Excel 
spreadsheet  created  by  Professor  Susan  Sanchez  (Sanchez,  2005).  The  NOLH 
design  considers  18  factors  with  129  levels  for  each  factor.  Figure  9  is  an  extract 
of  the  NOLH  design  that  depicts  the  first  50  of  the  129  design  points. 


36 


low  level 

0 

0 

0 

0 

0 

0 

0 

108 

72 

17 

7 

0.6 

0.6 

0.3 

high  level 

10 

100 

100 

100 

100 

100 

100 

180 

117 

42 

24 

1 

1 

1 

decimals 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

1 

1 

[1] 

[2] 

[3] 

[4] 

[5] 

[6] 

[7] 

[8] 

[9] 

[10] 

[11] 

[12] 

[13] 

[14] 

Comms 

Comms 

MAE 

LAE 

MAE  Re- 

LAE  Re- 

Air-space 

MAE 

LAE 

MAE  Detect 

LAE  Detect 

MAE  Class 

LAE  Class 

LAE  Class 

factor  name 

Latency 

Reliability 

Reactivity 
to  Inorgan 
SA 

Reactivity 
to  Inorgan 
SA 

directivity 
to  New 
Sighting 

directivity 
to  New 
Sighting 

Separation 

Airspeed 

Airspeed 

Range 

Range 

to  Detect 
Range 
Ratio 

to  Detect 
Range 
Ratio 

Prob 

2 

45 

45 

34 

69 

56 

76 

148 

114 

36 

20 

0.8 

1 

1 

9 

30 

46 

9 

45 

42 

20 

142 

89 

40 

17 

0.8 

1 

0.8 

4 

76 

27 

41 

16 

75 

65 

152 

98 

29 

10 

0.8 

0.9 

1 

7 

89 

37 

44 

76 

7 

41 

127 

93 

27 

13 

0.7 

1 

0.8 

0 

39 

23 

10 

53 

36 

93 

180 

100 

20 

24 

0.9 

0.7 

0.6 

7 

43 

1 

40 

33 

59 

4 

126 

77 

21 

24 

0.9 

0.8 

0.5 

4 

100 

29 

16 

23 

24 

94 

177 

105 

35 

9 

0.6 

0.6 

0.6 

6 

70 

6 

34 

92 

87 

2 

130 

81 

29 

8 

0.7 

0.7 

0.5 

0 

5 

20 

20 

98 

73 

43 

160 

110 

36 

12 

0.8 

0.7 

1 

10 

8 

27 

12 

10 

20 

55 

122 

78 

29 

9 

0.9 

0.6 

0.9 

0 

98 

28 

31 

7 

77 

22 

152 

109 

21 

16 

0.7 

0.7 

0.8 

9 

98 

38 

14 

100 

51 

91 

128 

86 

23 

18 

0.7 

0.8 

0.9 

5 

26 

18 

41 

72 

5 

36 

155 

112 

17 

11 

1 

0.9 

0.5 

8 

22 

35 

9 

23 

91 

54 

123 

90 

17 

12 

0.9 

1 

0.6 

3 

52 

23 

47 

21 

3 

14 

179 

108 

38 

20 

0.7 

0.8 

0.5 

8 

75 

3 

17 

52 

98 

73 

125 

91 

41 

17 

0.7 

0.8 

0.5 

2 

18 

74 

4 

70 

69 

49 

123 

99 

33 

17 

0.7 

1 

0.3 

10 

35 

78 

42 

20 

15 

48 

161 

74 

32 

19 

0.8 

0.8 

0.4 

2 

77 

52 

21 

3 

68 

53 

125 

113 

25 

15 

0.9 

1 

0.3 

6 

97 

75 

13 

80 

34 

16 

147 

83 

18 

15 

0.9 

0.8 

0.3 

3 

20 

95 

49 

66 

30 

100 

115 

101 

21 

24 

0.6 

0.8 

0.9 

6 

27 

92 

38 

13 

86 

27 

178 

97 

18 

17 

0.6 

0.7 

0.7 

2 

72 

98 

33 

34 

5 

95 

111 

111 

37 

8 

1 

0.8 

0.9 

7 

62 

98 

19 

63 

81 

28 

179 

90 

33 

9 

1 

0.7 

0.8 

3 

23 

61 

32 

99 

47 

26 

140 

109 

31 

14 

0.7 

0.6 

0.3 

9 

1 

57 

15 

29 

38 

81 

153 

85 

33 

13 

0.6 

0.7 

0.6 

2 

71 

100 

5 

12 

60 

30 

129 

110 

29 

19 

0.8 

0.7 

0.6 

10 

94 

70 

16 

75 

28 

59 

178 

74 

24 

21 

0.8 

0.6 

0.7 

3 

45 

55 

36 

54 

0 

38 

131 

93 

18 

8 

0.6 

0.9 

0.8 

6 

46 

70 

48 

42 

74 

91 

149 

88 

25 

10 

0.7 

0.9 

1 

5 

73 

76 

7 

39 

4 

1 

117 

111 

40 

21 

0.8 

0.9 

0.7 

5 

63 

89 

13 

73 

66 

77 

164 

73 

35 

19 

0.9 

0.9 

0.8 

1 

36 

34 

55 

49 

91 

56 

173 

95 

39 

20 

0.8 

0.9 

0.8 

9 

48 

9 

54 

40 

38 

3 

132 

96 

40 

23 

0.8 

0.9 

0.9 

4 

93 

41 

73 

15 

94 

82 

146 

91 

23 

13 

0.7 

0.9 

0.7 

5 

87 

44 

63 

95 

11 

13 

137 

108 

24 

9 

0.7 

0.8 

0.8 

1 

32 

10 

77 

84 

48 

66 

150 

72 

22 

20 

0.9 

0.7 

0.6 

8 

15 

40 

99 

30 

59 

61 

114 

107 

21 

19 

1 

0.7 

0.6 

3 

95 

16 

71 

5 

22 

85 

157 

76 

32 

14 

0.8 

0.8 

0.4 

9 

84 

34 

94 

86 

99 

37 

119 

107 

34 

16 

0.7 

0.7 

0.4 

3 

49 

20 

80 

68 

55 

10 

169 

83 

39 

8 

1 

0.7 

0.9 

8 

38 

12 

73 

52 

2 

71 

121 

104 

39 

10 

0.9 

0.6 

0.7 

5 

67 

31 

72 

27 

90 

11 

168 

87 

26 

20 

0.7 

0.8 

1 

6 

81 

14 

62 

74 

23 

92 

129 

102 

19 

21 

0.7 

0.8 

0.7 

2 

4 

41 

82 

56 

29 

55 

174 

87 

22 

8 

1 

0.9 

0.4 

9 

42 

9 

65 

2 

52 

67 

138 

110 

26 

10 

0.8 

1 

0.5 

0 

79 

47 

77 

13 

17 

16 

168 

73 

32 

15 

0.6 

1 

0.4 

6 

88 

17 

97 

78 

70 

98 

141 

105 

32 

18 

0.8 

0.9 

0.4 

5 

41 

96 

74 

96 

88 

95 

137 

83 

41 

14 

0.7 

0.9 

0.4 

8 

9 

58 

78 

41 

16 

40 

176 

109 

31 

17 

0.6 

0.9 

0.5 

Figure  9.  Nearly  Orthogonal  Latin  Hypercube  of  the  Design  of  Experiment 


Figure  10  plots  the  design  points  derived  from  the  18  factor  NOLH  crossed 
with  a  single-factor,  five-level  design  for  the  number  of  LAEs  to  each  MAE  to 
yield  a  total  of  645  design  points.  An  individual  dot  on  each  grid  represents  a 
design  point  with  levels  corresponding  to  the  factors  denoted  on  the  row  and  the 
column.  The  pairwise  scatter  plot  shows  good  space  filling  properties.  The  design 
points  are  highly  orthogonal  (all  correlation  values  lower  than  0.05),  with  the 
exception  of  the  LAE  classification  and  detection  range  pair,  the  MAE 
classification  and  detection  range  pair,  as  well  as  the  LAE  and  MAE  classification 
probability  pair.  These  pairs  of  factors  are  deliberately  constrained  such  that  one 
factor  is  expressed  as  a  random  fraction  of  the  other  for  every  design  point  while 

creating  the  NOLH.  For  instance,  the  LAE  classification  range  is  restricted  to  a 
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varying  fraction  of  the  LAE  detection  range  since  it  is  illogical  to  have  the  former 
parameter  larger  than  the  latter  for  the  same  sensor  system  on  an  LAE.  The  645 
design  points  are  run  for  50  replications  each,  which  results  in  a  total  of  32,250 
MANA  runs. 
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Figure  10.  Pairwise  Scatter  Plot  of  Design  Points  Using  a  Nearly  Orthogonal  Latin 

Hypercube  Crossed  With  a  Single-Factor 
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The  Tiller  is  a  software  tool  in  the  Project  Albert’s  toolkit  that  allows  the 
setup  of  XML  study  files  comprising  a  large  number  of  design  points  based  on 
the  experimental  design.  The  study  files  provide  the  sequence  of  changing 
values  for  the  associated  MANA  parameters,  thus  enabling  batch  processing 
without  the  need  to  load  each  simulation  run  manually.  The  study  files,  together 
with  the  base  case  simulation  model  file,  are  then  submitted  to  the  computing 
clusters  in  the  MAUI  High  Performance  Computing  Center  (MHPCC)  hosted  by 
the  Air  Force  Research  Laboratory  Center,  and  managed  by  the  University  of 
Hawaii  (Maui  High  Performance  Computing  Center,  2005). 
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IV.  DATA  ANALYSIS 


A.  DATA  PRE-PROCESSING 

The  output  data  generated  from  the  MHPCC  is  in  either  the  Comma 
Separated  Value  (CSV)  or  the  Microsoft  Data  Base  (MCB)  file  format.  The  data 
file  contains  the  excursion  number  for  each  design  point  and  a  random  index 
representing  the  replication  number  in  each  design  point,  as  well  as  the  settings 
for  factors  that  vary  in  the  NOLH  design.  Each  row  in  the  data  file  includes  the 
number  of  Blue  and  Red  killed,  the  numbers  of  agents  injured  and  killed  per 
squad,  the  time  steps  for  completing  the  simulation  run,  and  a  few  other  values. 
The  only  data  of  interest  in  the  output  files  are  the  total  number  of  Red  killed  and 
those  relating  to  squads  corresponding  to  the  TCTs  and  armor  battalions,  and 
obviously,  those  parameter  values  that  are  varied  in  each  simulation  run.  These 
data  are  then  imported  into  the  JMP  IN  Statistical  Software  (JMP  Statistical 
Discovery  Software  website,  2005)  on  which  data  analysis  is  performed.  The 
JMP  Statistical  Software  provides  many  valuable  analytical  tools  with  a  superior 
graphical  user  interface  so  there  is  a  relatively  short  learning  curve  in  using  the 
basic  features  sufficient  for  most  analytical  use.  The  software  is  also  selected  for 
its  powerful  data  pre-processing  capability  and  its  ability  to  handle  extremely 
large  data  sets. 

The  number  of  Red  agents  killed  represents  the  number  of  Red  targets 
that  are  classified  by  the  Blue  UAVs  at  the  end  of  the  simulation  run.  The 
expected  proportion  of  targets  classified  for  each  target  type  is  derived  by 
summarizing  the  data  over  the  replications  for  each  design  point  and  dividing  the 
mean  number  of  targets  classified  by  their  start-off  total.  These  expected 
proportions  of  targets  classified  provide  the  MOEs  for  analyzing  the  effectiveness 
of  the  cooperative  sensing  network  and  the  effects  the  factors  in  consideration 
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have  on  such  sensing  capability.  The  MANA  parameters  are  also  transformed 
back  to  their  real-world  representations  before  analyzing  the  data  in  order  to 
ease  interpretation  of  the  results. 


B.  ANALYSIS  TOOLS  AND  METHODOLOGIES 

Several  analytical  tools  and  methodologies  are  applied  to  the  model 
outputs.  These  sift  out  trends  intrinsic  within  the  data  to  help  answer  the 
questions  of  interest  and  provide  useful  insights  and  operational  guidance 
relevant  to  the  employment  of  cooperative  sensing  for  military  ISR  purposes.  The 
paragraphs  below  give  brief  descriptions  of  the  main  techniques  used  throughout 
the  analysis.  These  techniques  should  by  no  means  be  used  in  exclusion  of  the 
others;  using  all  in  a  complementary  manner  can  provide  better  perspectives  and 
operational  insights. 

1.  Data  Partitioning 

Data  partitioning  is  a  form  of  exploratory  modeling  (sometimes  known  as 
data  mining).  It  is  a  process  of  exploring  a  large  amount  of  data,  usually  using  an 
automated  method,  to  find  patterns  and  discoveries.  Data  partitioning  is  used  to 
recursively  partition  a  data  set,  automatically  splitting  the  data  at  optimum  points 
to  maximize  the  difference  in  the  values  of  the  response  variables  between  the 
branches  of  the  split.  The  result  is  a  decision  tree  that  classifies  each  observation 
into  a  group  (Sail  et  al.,  2005).  The  technique  is  often  used  for  exploring 
relationships  without  having  a  good  prior  model.  It  can  handle  large  problems 
with  relative  ease  and  the  results  are  very  interpretable. 


2.  Multiple  Linear  Regression 

Multiple  Linear  Regression  is  the  technique  of  fitting  or  predicting  a 
response  variable  from  a  linear  combination  of  several  other  variables.  The  fitting 
principle  is  least  squares,  which  finds  a  line  through  the  data  points  that 
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minimizes  the  sum  of  squared  distances  to  the  line  of  fit  (Sail  et  al. ,  2005).  It  is 
not  the  intend  of  this  analysis  to  use  the  regression  technique  in  the  conventional 
way  of  deriving  a  mathematical  model  for  predicting  the  proportion  of  Red  targets 
classified  given  a  set  of  cooperative  sensing  parameters;  the  agent-based 
distillation  models  are  not  set  up  for  such  details.  However,  the  technique  is  used 
to  provide  valuable  insights  into  the  factors  of  significance,  their  interactions  with 
one  another,  and  the  discovery  of  any  non-linearities  that  might  exist.  Pareto 
plots  are  often  used  to  visually  plot  bars  emphasizing  the  order  of  importance  of 
the  factors  from  the  most  importance  to  the  least.  The  prediction  profiler  in  JMP 
IN  presents  a  useful  interface  to  show  the  predicted  response  for  each 
combination  of  factor  settings.  It  provides  a  handy  way  to  look  at  the  effect  on  the 
predicted  response  of  changing  one  factor  setting  while  holding  the  others 
constant,  which  can  be  useful  for  judging  the  contribution  of  each  factor  to  the 
response. 

C.  RESULTS  OF  ANALYSIS 

1.  Sensing  Network  Configurations 

To  investigate  the  impact  the  number  of  LAEs  have  on  the  proportion  of 
targets  classified,  regression  analysis  is  performed  to  obtain  the  best  fit  for  the 
proportion  of  total  Red  targets,  TCTs  and  armor  targets  classified  against  the 
number  of  LAEs  respectively.  Refer  to  the  bivariate  fits  in  Section  1  of  Appendix 
B.  The  regression  models  show  that  the  quadratic  effects  are  significant  for  the 
proportions  of  total  Red  target  and  armor  classifications  (p-value  <  0.001),  but  not 
for  the  proportion  of  TCT  classification  (p-value  =  0.2870). 

For  ease  of  illustration,  the  proportions  of  targets  classified  are  grouped  by 
the  number  of  LAEs  to  obtain  the  average  proportions  of  targets  classified  for 
each  configuration  with  various  numbers  of  LAEs.  A  quadratic  curve  is  fitted  for 
the  proportion  of  total  Red  target  classifications  as  shown  in  Figure  1 1 ;  a  line  and 
a  curve  are  fitted  for  the  proportions  of  TCT  and  armor  classifications 
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respectively,  as  depicted  in  Figure  12.  An  initial  look  at  the  results  depicted  by 
the  curve  in  Figure  1 1  seems  to  suggest  that  one  MAE  and  four  LAEs  is  the  most 
cost-effective  cooperative  sensing  network  configuration  for  the  scenario  in 
consideration.  However,  the  detailed  breakdown  of  the  proportion  of  targets 
classified  into  the  types  of  targets  shown  in  Figure  12  reveals  that  while  the 
classification  rate  for  armor  targets  saturates  when  number  of  LAEs  reaches  four, 
the  classification  rate  for  TCT  targets  continues  to  increase  with  more  than  five 
LAEs.  TCTs  are  less  susceptible  to  UAV  detections  due  to  their  higher 
concealment  abilities.  Armor  battalions  have  strong  detection  signatures  and  are 
easily  classified  by  relatively  fewer  LAEs,  or  even  by  the  MAE  itself  which  has  a 
less  superior  classification  capability.  These  imply  that  an  investment  of  more 
sensing  resources  is  required  to  detect  TCTs.  Another  reason  for  the  differences 
in  the  number  of  LAEs  associated  with  diminishing  returns  could  be  due  to  the 
fleeing  characteristics  of  TCTs  and  their  ability  to  evade  detections,  thus  a 
denser  LAE  concentration  is  required  in  the  entire  area  of  operation  to  defeat 
their  counter-detection  capability  and  limited  windows  of  vulnerability.  One  would 
have  completely  missed  these  observations  without  looking  at  the  detailed  target 
classification  proportions  by  types.  This  illustrates  the  importance  of  defining  the 
correct  set  of  MOEs. 


Figure  1 1 .  Quadratic  Fit  for  the  Average  Proportion  of  Red  Targets  Classified  Against 

the  Number  of  LAEs 
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The  sudden  drop  in  the  proportion  of  armor  targets  classified  for  five  LAEs 
is  found  to  be  attributed  to  the  distribution  of  routes.  When  the  fifth  LAE  is 
introduced,  it  misses  most  of  the  armor  elements  in  Area  1  as  the  convoy 
proceeds  northward.  This  observation  further  demonstrates  the  sensitivity  of  the 
MOEs  to  variation  in  UAV  routings,  as  anticipated  during  the  model  development 
phase  and  discussed  in  Chapter  II. 

Bivariate  Fit  of  Mean(Prop  TCT  Class)  By  #  LAE  Bivariate  Fit  of  Mean(Prop  Armor  Class)  By  #  LAE 


Figure  12.  Linear  and  Quadratic  Fits  for  the  Average  Proportion  of  TCT  and  Armor 

Targets  Classified  Against  the  Number  of  LAEs 


The  histogram  plots  depicted  in  Figure  13  allow  further  exploration  of  the 
proportions  of  TCT  and  armor  targets  classified  for  each  MAE-LAE  configuration. 
These  reveal  high  frequencies  at  low  classification  rate  for  both  types  of  targets, 
even  though  the  means  and  the  modes  generally  increase  with  the  number  of 
LAEs.  Two  categorical  response  variables  are  defined  for  the  proportions  of  TCT 
and  armor  classification  lower  than  0.05  and  0.1  respectively.  A  logistic 
regression  model  is  fitted  for  each  of  these  response  variables  to  investigate  the 
rationales  for  the  low  classification  rates.  The  details  of  the  logistic  regression 
models  are  shown  in  Appendix  B.  The  analysis  identifies  LAE  classification  range 
and  LAE  responsiveness  to  MAE  cueing  as  common  factors  of  significance  that 
contribute  both  to  the  low  TCT  and  armor  classification  rates  amongst  few  other 
factors.  In  the  case  of  classifying  TCTs,  the  TCT  ability  to  conceal  and  its  sensor 
range  to  evade  UAV  detection  are  important  factors  explaining  the  low 
classification  rates.  The  partitioning  analysis  (refer  to  the  partitioning  tree  in 
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Appendix  B,  Section  1)  also  suggests  that  LAE  classification  ranges  less  than 
3,200  and  2,600  meters  can  only  classify  an  average  of  only  4%  and  6%  of  the 
total  TCT  and  armor  targets,  respectively. 
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13.  Distributions  of  the  Average  Proportion  of  TCT  (Top  Row)  and  Armor 
(Bottom  Row)  Targets  Classified  for  One  LAE  (Left)  to  Five  LAEs  (Right) 
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2. 


Factors  of  Significance 


The  Multiple  Linear  Regression  technique  is  used  to  determine  factors  that 
are  significant  to  the  proportion  of  targets  classified  during  the  four-hour  IPB  with 
the  cooperative  sensing  network.  Separate  analysis  are  conducted  for  the  TCT 
and  armor  targets,  as  distinctive  differences  exist  between  the  two  arrays  of 
targets  in  terms  of  their  target  profiles  and  operational  characteristics. 


Three  stepwise  linear  regression  models  are  fitted  for  the  proportion  of 
TCT  classified  with  consideration  of  simply  the  main  effects,  the  main  and 
quadratic  effects,  and  including  the  two-way  interactions  (Refer  to  Appendix  B  for 
the  detailed  models).  The  adjusted  R-square  values,  instead  of  the  standard  R- 
square  values,  are  considered  when  comparing  the  models  to  take  into  account 
the  number  of  terms  used  to  achieve  the  explanatory  power  of  the  models.  The 
quadratic  and  interaction  terms  in  the  full  model  account  for  approximately  20% 
of  explanatory  power  and  are  therefore  included  in  the  model  for  subsequent 
analysis.  There  are  seven  main  effects,  two  quadratics  and  five  interaction  terms 
in  the  selected  model  as  depicted  in  Figure  14.  These  fourteen  terms  explain 
73%  of  the  total  variability,  based  on  the  adjusted  R-square  value. 


Response  Prop  TCT  Class 
Summary  of  Fit 

RSquare  0.731611 

RSquare  Adj  0.725647 

Root  Mean  Square  Error  0.030259 

Mean  of  Response  0.067764 

Observations  (or  Sum  Wgts)  645 

Analysis  of  Variance 

Source  DF  Sum  of  Squares  Mean  Square  F  Ratio 

Model  14  1.5723599  0.112311  122.6672 

Error  630  0.5768141  0.000916  Prob  >  F 

C.  Total  644  2.1491740  <.0001 

Parameter  Estimates 


Term 

Intercept 

LAE  React  InOrg  SA 
MAE  Class  Range 
LAE  Class  Range 
LAE  Class  Prob 
TCT  Conceal 
TCT  Detect  Range 
#  LAE 

(LAE  React  InOrg  SA-0.00031)*(I_AE  Class  Range-3569.77) 
(LAE  Class  Range-3569.77)*(#  LAE-3) 

(LAE  Class  Prob-0.65039)*(#  LAE-3) 

(TCT  Conceal-50.0155)*(TCT  Detect  Range-5.08527) 

(TCT  Conceal-50.01 55)*(#  LAE-3) 

(LAE  Class  Range-3569.77)*(LAE  Class  Range-3569.77) 
(TCT  Conceal-50.01 55)*(TCT  Conceal-50.01 55) 


Estimate  Std  Error 
-0.062471  0.007717 
0.0120452  0.002049 
0.0000039  6.059e-7 
0.0000212  9.586e-7 
0.0503922  0.005722 
-0.000549  0.000041 
-0.003615  0.000397 
0.0198858  0.000842 
0.0000076  0.000002 
0.000006  6.537e-7 
0.0137356  0.004044 
-0.000093  0.000016 
-0.000156  0.000029 
-7.01 3e-9  7e-10 

-0.000008  0.000002 


t  Ratio 
-8.10 
5.88 
6.50 
22.09 
8.81 
-13.40 
-9.11 
23.60 
4.38 
9.19 
3.40 
-5.85 
-5.39 
-10.02 
-5.05 
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Figure  14.  Multiple  Linear  Regression  Model  for  the  Proportion  of  TCT  Classification 
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The  Pareto  plot  in  Figure  15  ranks  the  factors  in  the  full  model  in 
descending  order  of  importance.  The  number  of  LAEs  is  listed  as  having  the 
greatest  impact  on  TCT  classification  rate.  The  prediction  profiler  in  Figure  16 
highlights  a  substantial  increase  in  the  proportion  of  TCTs  classified  -  from  0.047 
to  0.126  (approximately  2.5  times)  -  as  the  number  of  LAEs  in  the  sensing 
network  increases  from  one  to  five  with  all  the  other  factors  held  constant. 


Pareto  Plot  of  Transformed  Estimates 

Term 

Orthog  Estimate 

#  LAE 

0.0281228 

LAE  Class  Range 

0.0240802 

TCT  Conceal 

-0.0155919 

(LAE  Class  Range-3569.77)*(LAE  Class  Range-3569.77) 

-0.0134044 

(LAE  Class  Range-3569.77)*(#  LAE-3) 

0.0110785 

TCT  Detect  Range 

-0.0108626 

LAE  Class  Rob 

0.0104792 

(TCT  Conceal-50.0155)*(TCT  Detect  Range-5.08527) 

-0  0082651 

LAE  React  InOrg  SA 

0.0068911 

MAE  Class  Range 

0.0068698 

(TCT  Conceal-50.01 55)*(#  LAE-3) 

-0.0064212 

(TCT  Conceal-50.01 55)*(TCT  Conceal-50.01 55) 

-0.0060193 

(LAE  React  InOrg  SA-0.00031)*(LAE  Class  Range-3569.77) 

0.0060000 

(LAE  Class  Prob-0.65039)*(#  LAE-3) 

0.0041515 

Figure  15.  Pareto  Plot  of  Factors  Significant  to  the  Classification  of  TCTs 
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Figure  16.  Prediction  Profiler  Showing  Approximately  2.5  Times  Improvement  in  TCT 
Classification  By  Increasing  the  Number  of  LAEs  From  One  to  Five 


It  is  also  apparent  that  improving  the  LAE  sensor  qualities  (from  the 

aspects  of  classification  range  and  classification  probability)  contributes 

considerably  to  the  increase  in  the  overall  effectiveness  of  classifying  the  TCTs. 
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The  MAE  is  generally  inferior  in  its  ability  to  classify  targets  due  to  the  high  flight 
altitude  and  wide  sensor  FOV  in  order  to  satisfy  the  wide  area  coverage 
requirement.  In  contrast,  the  LAEs  have  lower  flight  altitudes  and  more  focused 
sensors  that  enable  them  to  see  targets  with  better  resolutions,  so  though  LAEs 
are  myopic  in  their  coverage,  they  have  much  better  chances  of  classifying 
targets.  The  overall  classification  effectiveness  of  the  cooperative  sensing 
network  increases  when  the  LAEs  are  more  responsive  to  the  MAE  cueing 
instead  of  following  their  pre-assigned  routes.  The  interaction  plot  in  Figure  17, 
however,  shows  an  adverse  effect  when  the  LAEs  have  short  classification 
ranges  while  being  responsive  to  the  MAE  cueing.  An  LAE  with  small 
classification  coverage,  on  being  cued  by  the  MAE,  might  take  so  long  to  fly 
close  towards  the  TCT  that  it  will  miss  its  limited  vulnerability  window  and  lose 
the  target  completely.  Even  if  the  LAE  arrives  before  the  TCT  again  takes  cover, 
a  short  classification  range  means  the  LAE  has  a  limited  number  of  classification 
opportunities,  hence  a  low  chance  of  successfully  classifying  the  contact  during 
its  time  on  target.  This  wasteful  trip  can  cost  the  LAE  classification  opportunities 
which  would  have  arisen  if  it  kept  following  its  pre-assigned  route.  The  results 
also  show  that  extending  the  MAE  classification  range  has  a  small  but  positive 
impact  in  the  performance.  This  increases  the  number  of  classification 
opportunities  during  the  MAE’s  time  on  target  and  helps  compensate  for  the  poor 
classification  probability  per  attempt.  Note  that  the  benefits  of  having  higher 
classification  probabilities  for  LAEs  are  greater  when  there  are  more  LAEs  than 
when  there  are  fewer. 

The  negative  coefficients  associated  with  the  TCT  concealment  and  TCT 
detection  range  factors  present  the  obvious  case  that  TCTs  which  are  well 
concealed  with  longer  counter-detection  sensor  ranges  are  better  able  to  evade 
incoming  UAVs  and  avoid  detection  and  classification.  However,  the  interaction 
profiler  (Figure  16)  points  out  that  the  drop  in  the  proportion  of  classification  with 
increasing  TCT  concealment  ability  is  more  substantial  for  larger  number  of 
LAEs.  On  top  of  that,  it  is  also  observed  that  enhancing  the  two  TCT’s 
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parameters  simultaneously  has  a  synergistic  effect  in  reducing  their  susceptibility 
to  UAV  classification.  This  makes  perfect  sense,  as  it  is  not  helpful  for  a  TCT  to 
be  forewarned  by  its  counter-detection  sensor  of  an  approaching  UAV  if  it  does 
not  have  an  effective  concealment  capability. 
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Figure  17.  Interaction  Plot  of  Factors  Significant  to  the  Classification  of  TCTs 

Figure  18  shows  the  partitioning  tree  for  the  proportion  of  TCTs  classified. 

The  number  of  LAEs  is  the  first  break  point  for  this  model,  which  indicates  the 

number  of  LAEs  has  a  substantial  impact  on  effectiveness;  this  is  consistent  with 

the  results  presented  from  the  regression  analysis.  The  tree  displays  a  mean 

proportion  of  TCT  classification  of  only  4.5%  with  fewer  than  four  LAEs  and 

10.2%  with  four  or  more  LAEs  (refer  to  Appendix  B  for  the  full  partitioning  tree). 

Moving  down  the  right  branch  (i.e.,  using  four  or  more  LAEs),  only  3.8%  of  the 

TCTs  are  classified  if  the  LAE  classification  range  is  less  than  2,866  meters.  So  it 

is  futile  to  invest  in  more  LAEs  alone  but  it  is  important  to  invest  in  more  LAEs 

that  have  sensors  with  longer  classification  ranges.  Note,  however,  that  despite 
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the  investment  one  might  devote  into  the  cooperative  sensing  network,  the 
increment  in  classification  effectiveness  against  very  well-concealed  TCTs  is 
negligible. 


Figure  18.  Right  Branch  of  the  Partitioning  Tree  for  Proportion  of  TCT  Classification 

With  Four  or  More  LAEs 


While  partitioning  trees  are  useful  for  exploring  relationships  without 
having  a  good  prior  model  and  inherently  a  powerful  tool  for  capturing 
interactions,  McMindes  (2005)  highlights  a  few  drawbacks  associated  to  such 
form  of  exploratory  modeling.  There  is  no  sense  of  relative  importance  between 
the  factors  other  than  the  hierarchical  representation  of  the  tree  structure.  For 
instance,  Figure  18  shows  the  number  of  LAEs  is  more  important  than  LAE 
classification  range,  but  how  much  more  important?  There  is  also  no  sense  of 
indication  of  the  sensitivity  of  a  split.  The  LAE  classification  range  is  best  split  at 
2,866  meters,  which  is  something  absolutely  not  controllable  in  practice.  So  how 
much  is  lost  if  the  splitting  is  shifted  to  the  left  or  right  by  50  meters?  The  break 
points  also  tend  to  be  somewhat  unstable,  especially  as  the  groups  become 
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small.  For  these  reasons,  partitioning  trees  are  used  in  conjunction  with  results 
from  other  techniques,  such  as  Multiple  Linear  Regression,  and  to  provide 
qualitative  insights. 


A  similar  stepwise  linear  regression  approach  is  again  conducted  for  the 
proportion  of  armor  targets  classification.  Three  models  were  constructed  using 
three  sets  of  potential  terms:  the  first  model  involved  only  the  main  effects,  the 
second  model  involved  both  main  and  quadratic  effects,  and  the  third  (full)  model 
also  included  the  two-way  interactions.  Here,  the  full  model  explains  80%  of  the 
total  variability  while  the  quadratic  model  already  explains  a  notable  77%.  The 
interaction  terms  account  for  less  than  4%  of  the  explanatory  power  in  the  full 
model,  so  the  simpler  model  is  chosen  for  subsequent  analyses  (refer  to 
Appendix  B  for  the  detailed  models).  The  seven  main  effects  and  two  quadratic 
terms  in  the  selected  model  are  highlighted  in  Figure  19.  These  nine  terms 
explain  77%  of  the  total  data  variability. 


Response  Prop  Armor  Class 
Summary  of  Fit 


RSquare  0.777112 

RSquare  Adj  0.773953 

Root  Mean  Square  Error  0.052919 

Mean  of  Response  0.192601 

Observations  (or  Sum  Wgts)  645 

Analysis  of  Variance 


Source  DF  Sum  of  Squares  Mean  Square  F  Ratio 
Model  9  6.1999396  0.688882  245.9965 

&ror  635  1.7782373  0.002800  Prob  >  F 

C.  Total  644  7.9781769  <0001 


Parameter  Estimates 


Term 

Intercept 

LAE  React  InOrg  SA 
LAE  React  En  Contact 
LAEAirspeed 
MAE  Class  Range 
LAE  Class  Range 
LAE  Class  Rob 
#  LAE 

(LAE  Class  Range-3569. 77)*(LAE Class  Range-3569.77) 
(#  LAE-3)*(#  LAE-3) 


Estimate 

Std  Error 

t  Ratio 

Rob>|t| 

-0.218018 

0.019501 

-11.18 

<.0001 

-0.014634 

0.003583 

-4.08 

<.0001 

0.0154657 

0.003583 

4.32 

<.0001 

0.000569 

0.000088 

6.45 

<.0001 

0.0000054 

0.000001 

5.14 

<.0001 

0.0000567 

0.000002 

33.85 

<.0001 

0.0964538 

0.010006 

9.64 

<.0001 

0.0310841 

0.001473 

21.10 

<.0001 

-2.196e-8 

1.187e-9 

-18.49 

<.0001 

-0.022495 

0.001245 

-18.06 

<.0001 

Figure  19.  Multiple  Linear  Regression  Model  for  the  Proportion  of  Armor 

Classification 


The  LAE  classification  range  and  the  number  of  LAEs  are  ranked  as  the 

two  most  important  factors  that  affect  the  proportion  of  armor  classification,  as 
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the  Pareto  plot  in  Figure  20  illustrates.  The  quadratics  of  the  two  main  effects  are 
next  on  the  list,  and  indicate  strong  non-linear  effects;  the  negative  regression 
coefficients  for  the  quadratic  terms  with  positive  corresponding  main  effect 
coefficients  indicate  that  increasing  either  the  LAE  classification  range  or  the 
number  of  LAEs  will  eventually  lead  to  diminishing  returns.  This  is  illustrated 
clearly  in  the  prediction  profiler  graphs  in  Figure  21.  In  fact,  the  proportion  of 
armor  classification  shoots  up  from  0.053  to  0.311  (almost  a  fivefold 
improvement)  as  the  LAE  classification  range  extends  from  1,433  to  4,900 
meters,  and  enlarging  the  LAE  classification  footprint  beyond  this  upper  limit 
reduces  the  overall  classification  effectiveness  of  the  sensing  network.  Visually 
watching  a  few  simulation  runs  suggests  why  this  occurs.  An  LAE  with  long 
classification  range  is  easily  distracted  by  the  surrounding  neutral  vehicles  within 
the  sensor  footprint,  and  this  renders  the  LAE  ineffective  in  its  mission  to  search 
out  the  Red  targets.  As  discussed  in  the  earlier  section,  the  drop  in  the  proportion 
of  armor  classification  for  five  LAEs  is  attributed  to  the  distribution  of  routes  when 
the  fifth  LAE  is  introduced,  since  these  routes  miss  most  of  the  armor  elements  in 
Area  1 . 


Pareto  Plot  of  Transformed  Estimates 


Term 

Orthog  Estimate 

LAE  Class  Range 

0.0631298 

#  LAE 

0.0439596 

(LAE  Class  Range-3569.77)*(LAE  Class  Range-3569.77) 

-0.0385286 

(#  LAE-3)*(#  LAE-3) 

-0.0376414 

LAE  Class  Prob 

0.0192751 

LAE  Airspeed 

0.0134963 

LAE  React  En  Contact 

0.0104379 

LAE  React  InOrg  SA 

-0.0086792 

MAE  Class  Range 

0.0074305 

Figure  20.  Pareto  Plot  of  Factors  Significant  to  the  Classification  of  Armor  Targets 
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LAEReact  LAEReact  LAE  MAEOass  LAEOass  LAECIass  #  LAE 

InOrg  SA  En  Contact  Airspeed  Range  Range  Rob 

Figure  21 .  Prediction  Profiler  Showing  Almost  5  Times  Improvement  in  Armor 
Classification  By  Increasing  LAEs  Classification  Range  Less  Than  3.5  Times 

As  in  the  case  of  TCT  classification,  better  quality  LAE  sensors  (in  terms 
of  higher  classification  probabilities)  enhance  the  classification  effectiveness 
against  the  armor  targets.  Increasing  the  MAE  classification  range  leads  to  a 
marginal  improvement  on  the  classification  effectiveness  as  well,  perhaps 
because  it  also  increases  the  number  of  classification  opportunities.  In  contrast, 
increasing  the  MAE  classification  probability  was  not  shown  to  have  an  impact. 
The  probability  of  an  MAE  successfully  classifying  an  intended  target  in  each 
opportunity  is  too  low  to  make  significant  contribution  to  the  overall  effectiveness 
even  at  the  highest  range,  and  therefore  not  a  significant  factor.  To  find  more  of 
the  highly  mobile  armors  that  are  commonly  deployed  in  a  formation,  a  faster 
LAE  does  contribute  to  a  higher  classification  success.  This  is  not  true  for 
detecting  the  TCTs  in  this  scenario,  as  they  generally  move  over  shorter 
distances. 

Unlike  classifying  TCTs,  an  LAE  should  lean  towards  being  more 
responsive  to  follow  a  newly  classified  armor  element  since  the  discovery  of  one 
is  likely  to  lead  to  the  entire  formation.  Such  LAE  behaviors  would  not  benefit 
TCT  classification  since  they  are  mostly  sparsely  deployed  in  isolation.  Being 
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responsive  to  the  MAE  cueing,  in  this  case,  may  result  in  lower  proportion  of 
armor  classification,  but  higher  TCT  discovery. 

The  partitioning  tree  for  the  proportion  of  armor  classified  (Figure  22) 
shows  LAE  classification  range  to  be  the  first  break  point  for  this  model,  which  is 
consistent  with  the  results  presented  from  the  regression  analysis.  The  tree 
presents  a  mean  proportion  of  armor  classification  of  23.2%  when  the  LAE 
classification  range  is  at  least  2,579  meters  and  5.8%  below  that.  The  next  two 
levels  of  partitioning  are  based  on  the  number  of  LAEs  on  both  branches,  thus 
indicating  a  non-linear  relationship  between  the  number  of  LAEs  and  the 
response  variable  with  the  best  values  around  three  to  four  LAEs.  As  discussed 
earlier,  the  introduction  of  the  fifth  LAE  misses  most  of  the  armor  elements  in 
Area  1  due  to  its  pre-assigned  routing.  A  faster  LAE  could,  however,  mitigate  this 
by  reaching  the  area  of  responsibility  earlier  before  the  armor  battalion  moves 
away. 


Figure  22.  Partitioning  T ree  for  Proportion  of  Armor  Classification 


3.  UAV  Behaviors 

The  contour  plots  in  Figure  23  depict  the  contours  for  the  proportions  of 
TCT  and  armor  classification  with  respect  to  the  LAE  responsiveness  toward  new 
enemy  contacts  they  make  (labeled  “LAE  React  En  Contact”  on  the  Y-axis)  and 
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MAE  redirections  (labeled  “LAE  React  Inorg  SA”  on  the  X-axis).  The  dark  blue 
patches  represent  higher  proportions  of  classification  in  contrast  to  the  light  blue 
patches.  The  points  on  the  plots  indicate  combinations  of  these  two  factors  that 
appeared  among  the  experiments. 

The  plot  shows  a  higher  concentration  of  dark  blue  patches  on  the  right 
half  of  the  plot,  thus  suggesting  better  classification  effectiveness  against  the 
TCTs  when  the  LAEs  have  better  responsiveness  towards  the  MAE  cueing. 
However,  LAE  response  to  a  newly  classified  enemy  contact  does  not  seem  to 
matter  in  achieving  better  overall  classification  performance.  The  result  is  intuitive 
since  the  TCTs  are  mostly  sparsely  distributed  in  this  scenario.  It  is  more  likely 
that  they  are  first  detected  by  the  MAE  with  the  wide  area  sensor  which  then 
cues  the  LAEs  to  perform  a  close-in  classification  of  these  targets.  Since  the 
TCTs  are  sparsely  located,  having  the  LAEs  follow  newly  discovered  contacts  is 
unlikely  to  lead  to  another  TCT  detection. 


Figure  23.  Contour  Plots  Showing  Proportions  of  TCT  (Left)  and  Armor  (Right) 
Classification  With  Respect  to  LAE  responsiveness  Toward  New  Enemy 
Contacts  and  MAE  Redirections  [Best  Viewed  in  Color] 


For  classifying  armor  targets,  the  dark  blue  patches  clustering  around  the 
top  left  of  the  plot  suggest  that  having  LAEs  that  are  more  responsive  towards  a 
newly  classified  enemy  contact  and  less  towards  the  MAE  cueing  significantly 
improves  the  overall  classification  effectiveness  against  armor  targets.  As  armor 

elements  tend  to  appear  in  formations,  following  a  newly  classified  armor 
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element  is  likely  to  lead  to  the  discovery  of  more  armor  elements  in  close 
neighborhood.  However,  once  an  LAE  has  found  a  cluster  of  targets,  being  too 
responsive  to  the  MAE  cueing  might  distract  the  LAE  from  making  its  kill  and 
result  in  a  lower  overall  proportion  of  armor  classification. 

4.  Impact  of  LAE  Failure 

To  examine  how  an  LAE  failure  during  the  four-hour  IPB  operation  might 
affect  the  overall  classification  performance  -  and  how  a  cooperative  sensing 
network  could  mitigate  this  performance  degradation  -  the  scenario  with  four 
LAEs  is  modified  slightly  so  that  one  LAE  is  randomly  selected  to  fail.  Successive 
sets  of  experiments  are  conducted  where  this  LAE  fails  immediately  from  the 
start  of  the  scenario,  and  at  the  first,  the  second,  and  the  third,  and  the  last  hour 
of  the  operation.  When  an  LAE  fails,  the  other  UAVs  continue  on  with  the  pre¬ 
assigned  routes  unless  otherwise  influenced  by  their  responses  to  UAV 
redirections;  there  is  no  re-planning  of  the  UAV  routes.  A  few  other  factor  ranges 
are  modified  based  on  the  results  described  earlier  in  this  chapter.  First,  only  two 
discrete  levels  are  used  for  link  reliability:  0  represents  no  cooperation  and  100 
represents  full  cooperation.  The  ranges  of  reactivity  to  other  UAV  SA  for  both  the 
MAE  and  LAEs  are  also  modified  to  take  on  values  between  0  to  +1.  This 
represents  that  the  UAVs  are  at  least  as  reactive  to  redirection  as  to  following 
their  pre-assigned  routes.  The  new  NOLH  is  then  crossed  with  the  LAE  lifetime 
(i.e.,  number  of  hours  before  a  randomly  selected  LAE  fails)  to  obtain  the  new 
experimental  design  matrix. 

The  graphs  in  Figure  24  show  the  average  proportions  of  TCT  and  armor 

classification  when  an  LAE  fails  at  various  time  instances  with  and  without 

cooperative  sensing  capability.  The  graphs  again  clearly  illustrate  that 

cooperation  does  help  in  classifying  TCTs  but  not  armor  targets.  However,  for 

both  target  types,  having  a  failed  LAE  significantly  degrades  classification 

performance  -  as  much  as  21%  in  the  four-hour  operation,  comparing  an  LAE 

failing  at  the  start  and  the  end  of  the  operation.  Although  the  improvement  gained 
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in  classification  proportion  for  TCTs  is  constant  regardless  of  the  time  of  LAE 
failure,  an  improvement  of  0.012  from  a  base  of  0.065  to  0.087  is  worth  noting. 


TCT 


Classification  Proportion  Against  LAE  Lifetime  With  and 
Without  Cooperative  Sensing 


-♦ —  No  Cooperation 
-■ —  Cooperation 


Armor  Classification  Proportion  Against  LAE  Lifetime  With 
and  Without  Cooperative  Sensing 


■  No  Cooperation 

■  Cooperation 


Figure  24.  Graphs  Depicting  Average  Proportions  of  TCT  and  Armor  Classification 
Against  LAE  Lifetime  With  and  Without  Cooperative  Sensing  Capability 


In  contrast,  the  gentler  slope  in  the  armor  classification  proportion  when 
the  LAE  lifetime  drops  from  four  to  zero  hours  shows  that  less  degradation  in 
classification  performance  occurs  when  the  UAVs  are  part  of  a  cooperative 
sensing  network.  This  indicates  that  although  cooperative  sensing  does  not  help 
in  the  overall  armor  classification  in  this  particular  scenario,  it  may  provide  a 
more  robust  solution  in  some  scenarios. 
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In  conjunction  with  the  earlier  observations,  one  should  note  that  while  it  is 
undesirable  for  an  LAE  to  be  responsive  to  MAE  redirection  for  a  while  once  an 
armor  entity  is  found,  but  instead  follow  the  new  contact  which  will  likely  lead  to 
the  discovery  of  more  armor  targets,  at  other  times  it  may  be  worthwhile  to  react 
to  an  MAE  redirection  before  the  LAE  has  even  made  its  initial  classification.  This 
suggests  that  appropriate  switching  between  these  behaviors  during  different 
phases  of  an  LAE  operation  is  necessary.  In  summary,  being  responsive  to  MAE 
cueing  may  not  always  give  the  best  overall  classification  results;  reacting  to  a 
cueing  request  may  lead  to  the  classification  of  other  potential  targets  but  risk  the 
opportunity  of  a  greater  payoff  by  remaining  in  its  path.  Prior  intelligence  of  the 
enemy’s  courses  of  action  and  deployments  is  important  for  deciding  appropriate 
redirections  of  LAEs  even  in  the  presence  of  cooperative  sensing.  Following  a 
fixed  set  of  standard  operating  tactics  regardless  of  the  targets  of  interest  and 
enemy’s  courses  of  action  is  not  an  effective  way  to  deploy  a  cooperative 
sensing  network. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  KEY  OPERATIONAL  INSIGHTS 

The  overall  effectiveness  of  the  cooperative  sensing  network  for 
classifying  TCTs  and  armor  targets  with  four  LAEs  stand  at  relatively  low  mean 
percentages  of  9.8%  and  27.2%,  respectively.  The  low  classification  proportions 
are  attributed  to  the  limited  time  frame  the  UAVs  have  for  the  IPB  operation  and 
the  modestly  large  number  of  neutral  vehicles.  Depending  on  the  setup  of  the 
UAV  behaviors  and  sensor  parameters,  the  UAVs  may  be  distracted  by  this 
clutter  around  interesting  targets,  thus  unable  to  complete  the  routes  they  were 
initially  designated.  However,  based  on  the  author’s  past  experiences  with 
intelligence  collection  experiments,  the  results  fall  within  the  respectable  range 
from  an  operational  standpoint,  given  the  short-range  optical  sensory  assets 
searching  in  a  considerably  large  area  of  operations.  The  relative  classification 
performance  of  approximately  one  TCT  to  three  armor  targets  makes  reasonable 
sense  as  the  armor  targets  have  unlimited  durations  of  vulnerability  and  the 
movement  of  an  armor  battalion  generally  emits  strong  signatures  which  leads  to 
easy  detection.  They  are  also  closely  clustered  which  means  the  acquisition  of 
one  tank  gives  away  the  others  in  the  formation.  However,  the  TCTs  are  direct 
opposites  in  their  operations;  they  are  well  concealed,  operate  individually  and 
have  limited  periods  of  vulnerability,  thus  making  them  hard  targets  to  catch. 

Two  out  of  the  seven  main  effects  filtered  out  as  factors  of  significance  by 
the  regression  analysis  describe  Red  TCT  characteristics  that  are  beyond  the 
control  of  Blue  forces.  These  two  factors  are  the  TCT  concealment  factor  which 
dictates  their  ability  to  conceal  themselves  to  avoid  UAV  detection  and 
classification,  and  the  counter-detection  sensor  range  which  determines  their 
ability  to  detect  approaching  UAVs  and  hence  evade  UAV  detection  and 
classification.  While  TCT  counter-detection  sensor  range  does  not  show  a  steep 
impact  on  the  UAV  classification  performance  -  and  can  be  overcome  with  better 

LAE  sensors  with  longer  classification  ranges  and  higher  classification 
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probabilities  -  well-camouflaged  TCTs  reduce  the  overall  UAV  classification 
capability  substantially.  This  suggests  that  mounting  an  EO  sensor  alone  on  the 
LAEs  is  not  an  effective  solution  to  the  problem  of  locating  well-camouflaged 
TCTs.  Perhaps  a  more  viable  option  for  classifying  TCTs  in  densely  forested 
areas  is  to  equip  the  LAEs  with  IR  sensors  as  well,  to  allow  the  targets  to  be 
tracked  based  on  heat  emissions  instead  of  solely  on  optical  detection.  In  this 
way,  the  classification  opportunities  are  no  longer  limited  to  the  windows  of 
vulnerabilities  when  the  TCT  appears. 

The  increase  in  the  LAE  responsiveness  to  an  MAE  redirection  versus  its 
tendency  to  follow  its  pre-assigned  routes  has  a  marginal  positive  effect  on  the 
proportion  of  TCT  classification  in  the  scenario.  The  MAE,  which  is  designed  to 
provide  global  initial  detection,  cues  all  the  LAEs  to  perform  a  close-in 
classification  when  it  finds  suspicious  contacts.  Once  the  first  LAE  to  arrive  has 
successfully  classified  the  contacts,  the  modeling  approach  used  in  this  thesis 
makes  certain  that  all  other  responding  LAEs  are  turned  back.  However,  such 
LAE  behavior  deters  effective  overall  classification  performance  on  armor 
targets.  Instead,  an  LAE  should  adopt  the  tactics  to  follow  a  newly  classified 
armor  contact  which  would  likely  lead  to  the  acquisition  of  more  armor  elements 
in  the  formation.  An  LAE  that  is  responsive  to  the  MAE  cueing  may  benefit  the 
first  successful  armor  classification.  However,  if  this  LAE  is  frequently  distracted 
by  subsequent  MAE  redirections,  and  the  associated  traveling  times  are  long 
because  the  cues  are  dispersed,  then  it  may  spend  time  traveling  between 
unclassified  targets  that  could  otherwise  be  more  effectively  used  to  acquire 
more  armor  elements  clustering  around  the  first. 

Communication  parameters  such  as  link  reliability  and  message  latency 
are  notably  absent  from  the  regression  models  that  depict  factors  of  significance 
in  a  cooperative  sensing  network  deployment.  The  author  opines  the  observation 
does  correctly  reflect  the  actual  environment  in  that  the  range  of  message 
latency  between  0  to  100  seconds  is  short  compared  to  the  LAE  flight  time  to  the 
suspicious  contact  in  response  to  an  MAE  redirection,  thus  does  not  impact  the 
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overall  classification  effectiveness  of  the  sensing  network.  The  relatively  low 
tempo  of  MAE  redirections  during  the  short  four-hour  IPB  operation  might  also 
give  the  MAE  sufficient  opportunities  for  retransmissions  even  in  the  presence  of 
poor  link  availability.  Note  also  that  the  airspace  separation  between  the  UAVs 
does  not  play  a  significant  role  in  determining  the  overall  classification 
proportions  of  the  targets.  The  separations  between  the  MAE  and  the  LAEs  in 
practice  are  not  a  concern  since  they  tend  to  fly  at  different  altitudes.  Similarly, 
the  separations  between  the  LAEs  are  of  little  concern  in  this  scenario  since  their 
flight  plans  have  already  ensured  no  two  LAEs  are  close  to  each  other  at  any 
point  in  time.  On  an  MAE  redirection,  all  other  responding  LAEs  are  turned  back 
after  the  first  LAE  to  arrive  has  successfully  classified  the  contact;  the  likelihood 
of  having  two  LAEs  closing  into  the  same  target  is  small.  It  is  obvious  in 
retrospect  that  airspace  separation  has  little  impact,  if  not  none  at  all,  on  this 
particular  sensing  network  performance. 

While  much  is  discussed  about  the  behaviors  of  the  UAVs  in  response  to 
the  presence  of  shared  SA  in  a  cooperative  sensing  environment  in  the  earlier 
observations,  many  of  the  factors  that  contribute  directly  or  indirectly  to  the  extent 
of  overall  coverage  of  the  sensing  network  appear  in  the  analysis  as  significant. 
Factors  such  as  MAE  classification  range,  LAE  classification  range  and  number 
of  LAEs  show  general  trends  of  improving  classification  capabilities  for  both 
target  types  as  their  values  increase.  However,  the  increase  in  classification 
proportion  for  increasing  LAE  classification  range  tapers  off,  and  even  tips 
downwards  around  the  5,000  meters  range.  In  this  scenario,  the  surrounding 
neutral  vehicles  easily  distract  an  LAE  with  a  large  classification  footprint.  This 
renders  the  LAE  ineffective  in  its  mission  to  search  out  the  Red  targets.  In 
addition,  the  overall  classification  performance  of  the  sensing  network  is  very 
sensitive  to  the  routings  of  the  UAVs,  which  suggests  that  it  is  worthwhile  to 
devote  much  time  and  effort  into  collecting  information  regarding  the  enemy’s 
target  profiles,  position  and  terrain  prior  to  preparing  the  UAV  search  paths  to 
provide  the  greatest  ground  coverage  in  areas  of  suspected  operation.  The 
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analysis  reveals  that  while  much  is  invested  into  netting  UAVs  together  for 
interaction  and  cooperation,  the  traditional  emphasis  of  investing  in  building 
better  quality  sensors  that  see  further  and  clearer  is  as  important. 

Cooperative  sensing  does  provide  a  more  robust  solution  in  this  scenario 
in  terms  of  the  classification  performance  should  an  LAE  fail  or  be  shot  down 
during  the  four-hour  operation.  Without  cooperation,  an  LAE  failure  will  result  in  a 
“hole”  in  the  ISR  coverage  of  the  entire  area  of  operation  unless  a  spare  LAE  is 
dispatched.  The  detrimental  effect  is  mitigated  to  a  certain  extent  with 
cooperative  sensing  in  that  another  LAE  may  be  redirected  to  patch  this  “hole.” 
However,  an  LAE  being  responsive  to  an  MAE  cueing  may  not  always  give  the 
best  overall  classification  results;  reacting  to  a  cueing  request  may  lead  to  the 
classification  of  other  potential  targets  but  risk  the  opportunity  of  a  greater  payoff 
by  remaining  in  its  path.  Prior  intelligence  of  the  enemy’s  courses  of  action  and 
deployments  is  important  for  deciding  appropriate  redirections  of  LAEs  even  in 
the  presence  of  cooperative  sensing.  Following  a  fixed  set  of  standard  operating 
tactics  regardless  of  the  targets  of  interest  and  enemy’s  courses  of  action  is  not 
the  way  forward  in  employing  a  cooperative  sensing  network. 


B.  RECOMMENDATIONS  FOR  FUTURE  WORK 

As  with  all  exploratory  investigations,  the  thesis  has  led  to  more  questions 
and  identified  other  relevant  areas  of  interest  for  follow-on  work.  The  ensuing 
paragraphs  recommend  some  possible  aspects  that  merit  further  research. 

The  analysis  highlights  the  sensitivity  of  the  models  and  the  MOEs  to 
variations  in  the  UAV  routings.  While  the  routings  in  this  research  are  based  on 
an  intuitive  justification  which  was  deemed  appropriate  and  sufficient  for  the 
study  by  subject  matter  experts,  other  routings  would  also  be  justifiable  with 
differing  operational  considerations  and  other  prior  intelligence  on  the  adversary 
courses  of  action,  targets  and  terrains.  Designing  an  experiment  that  explores 
various  routing  guidance  and  options,  based  on  a  broader  number  of  scenarios 
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with  possible  enemy  configurations  (deployment  plans  and  courses  of  action)  to 
plan  a  robust  set  of  routes,  could  yield  a  more  accurate  sense  of  the  design  of  a 
cooperative  sensing  network  and  a  robust  solution. 

Based  on  the  findings  from  the  analysis,  the  responsiveness  of  an  LAE  to 
an  MAE  redirection  should  vary  depending  on  the  current  status  and  recent  types 
of  targets  classified.  For  instance,  an  LAE  which  has  recently  made  an  armor 
classification  might  not  want  to  respond  to  an  incoming  MAE  cueing,  but  follow 
the  newly  classified  contact  which  is  likely  to  lead  to  more  armor  target 
acquisitions.  On  the  contrary,  an  idling  LAE  flying  on  its  designated  flight  plan  or 
one  that  has  just  made  a  TCT  classification  might  get  better  payoff  by  being  very 
responsive  to  an  MAE  redirection.  This  prompts  another  area  that  is  worth  further 
exploration:  using  an  agent-based  simulation  model  to  derive  more  “intelligent” 
means  of  cooperation,  taking  into  account  the  LAE  status  and  the  type  of  recent 
classification. 

Although  MANA  has  been  commonly  used  for  evaluating  ISR-related 
problems  in  a  number  of  studies,  such  as  those  presented  in  the  Background 
Work  Section  of  Chapter  I,  it  is  seldom  used  for  modeling  cooperative  sensing 
where  information  passing  across  a  communications  network  could  influence  the 
behaviors  of  the  agents.  There  is  currently  no  accurate  way  to  verify  the  models, 
except  by  visual  inspection  of  a  few  instances  of  the  model  and  by  the 
justification  of  the  set  of  average  classification  proportions  of  the  targets  falling 
within  ballpark  ranges.  Porting  the  models  over  to  another  agent-based  platform, 
like  PYTHAGORAS  (another  agent-based  distillation  model  in  the  Project  Albert 
suite  of  models),  would  be  beneficial  for  cross-validating  the  results  of  this 
analysis.  This  cross-validation  is  particularly  useful  to  ensure  that  certain 
complex  behaviors  are  not  modeled  incorrectly,  or  that  insights  obtained  from  the 
analysis  are  not  due  to  artifacts  of  a  specific  modeling  platform.  However,  one 
should  be  cautious  to  avoid  focusing  on  specific  numerical  comparisons  based 
on  the  model  output  data  since  different  simulations  yield  different  set  of  results. 
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Instead,  a  cross-validation  process  would  reveal  whether  the  two  sets  of  model 
outputs  yield  qualitatively  similar  insights. 

Another  advantage  of  porting  the  models  over  to  PYTHAGORAS  would  be 
to  leverage  the  power  and  versatility  of  its  sensor  setup.  An  agent  in 
PYTHAGORAS  may  carry  up  to  three  sensors,  and  each  sensor  may  have 
different  detection  parameters  with  different  detectability  on  various  terrain 
features  (Bitinas,  2004).  This  flexible  sensor  setup  enables  the  exploration  of  a 
cooperative  sensing  network  configured  for  multiple  sensors,  such  as  an  EO  and 
a  FOPEN,  on  a  UAV.  The  effects  of  having  a  mix  of  sensor-bearing  UAVs  in  the 
area  of  operation  could  also  be  explored.  The  inclusion  of  FOPEN  or  even  the 
multi-modal  SAR  sensors  could  facilitate  exploration  of  the  improvement  in  the 
overall  classification  performance  of  the  sensing  network.  However,  the 
downside  of  using  PYTHAGORAS  is  its  simplicity  in  modeling  communications 
with  the  rudimentary  broadcast  mechanism.  The  broadcast  mechanism  does  not 
permit  the  considerations  of  message  latency  and  link  reliability  as  implemented 
in  the  MANA  models,  but  simply  exchanges  SA  information  among  the  UAVs.  By 
studying  similar  models  in  MANA  and  PYTHAGORAS,  an  analyst  could  attempt 
to  exploit  the  strengths  of  each  specific  modeling  platform  to  gain  better 
understanding  of  how  a  broader  variety  of  sensors,  cooperative  information 
sharing,  and  UAV  tactics  influence  the  mission  effectiveness. 

To  gain  deeper  insights  on  the  effectiveness  of  a  cooperative  sensing 
network  for  ISR  operations,  scenarios  where  multiple  local  LAEs  operate 
independently  in  the  absence  of  a  coordinating  component  like  the  MAE  should 
be  investigated.  A  greater  challenge  is  modeling  the  effects  of  UAV  swarming, 
where  the  deployment  of  a  larger  concentration  of  small  UAVs  with  myopic 
sensors  operating  in  a  distributed  yet  cooperative  manner  could  be  explored. 
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c. 


SUMMARY 


The  success  of  a  battle  is  increasingly  reliant  on  more  accurate  and  timely 
collection  of  the  battle-space  intelligence.  Unmanned  systems,  and  certainly 
UAVs,  are  widely  used  in  today’s  military  ISR  operations  and  will  be  more 
prevalent  in  the  next  generation  military  as  war-fighting  concepts  such  as 
Network  Centric  Warfare  and  Distributed  Operation  emerge.  Tighter  integration 
and  tactics  development  to  enhance  cooperation  between  these  intelligence 
collection  assets  is  one  of  the  current  focuses  among  the  military  intelligence 
community,  and  the  motivation  for  this  thesis. 

The  use  of  agent-based  simulation  is  definitely  appropriate  to  seek 
insights  relating  to  the  tactics  and  behaviors  in  using  a  cooperative  sensing 
network  for  military  ISR.  The  models  are  easy  to  construct  and  consume 
relatively  less  computing  power  than  most  conventional  simulation  models.  They 
can  be  executed  for  tens  of  thousands  of  replications  to  exhaust  many  possible 
representations  of  real-life  situations.  By  applying  suitable  experimental  designs, 
one  can  move  from  a  realm  of  impossibility  when  the  number  of  variations  to  be 
investigated  is  insurmountably  large  to  a  problem  that  can  be  realistically  tackled 
within  days  or  weeks.  The  simulation  also  provides  ways  to  analyze  a  much 
wider  variety  of  settings  than  time  and  cost  would  permit  in  a  field  experiment. 
The  insights  from  agent-based  simulation  models  may  also  provide  guidance  for 
setting  up  future  higher-resolution  simulation  experiments  or  field  tests. 

In  this  thesis,  the  author  has  built  a  number  of  agent-based  models 

intended  to  capture  the  essential  details  of  UAV  performance  for  the  purpose  of 

answering  the  questions  that  might  be  of  interest  for  system  designers  and 

operators  to  implement  a  cooperative  sensing  network  for  the  military.  However, 

with  careful  considerations,  appropriate  assumptions  are  also  made  to  keep  the 

model  manageable  within  the  identified  scope.  The  data  collected  from  the 

simulations  are  analyzed  and  investigated  using  a  wide  array  of  complementary 

analytical  tools  and  techniques.  The  study  has  successfully  led  to  several 

operational  insights  pertaining  to  the  design  and  use  of  cooperative  sensing  for 
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ISR  purposes,  including  the  importance  of  having  good  UAV  sensor  capabilities 
and  “intelligent”  application  of  UAV  cooperation  tactics  based  on  the  interested 
target  characteristics,  to  achieve  high  overall  effectiveness.  The  study  has  also 
spawned  further  questions  and  identified  relevant  areas  for  follow-on  work. 
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APPENDIX  A.  UAV  ROUTINGS 


This  appendix  presents  snapshots  of  the  UAV  routings  for  each  MAE-LAE 
configuration  as  well  as  the  rationales  that  led  to  these  routings.  During  the 
development  of  these  routes,  the  author  consulted  with  Captain  Starr  King,  USN, 
NWDC-Sponsored  Chair  of  Warfare  Innovation  and  Chair  of  Applied  Systems 
Analysis  of  the  Naval  Postgraduate  School  and  Captain  Kevin  McMindes,  a 
USMC  pilot.  Both  consider  the  routings  reasonable  and  sufficiently  detailed  for 
the  purpose  of  modeling  and  analysis  to  answer  the  questions  of  interest. 
However,  one  should  not  preclude  other  alternative  routes  based  on  various 
operational  judgments  and  objectives. 
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1. 


MAE  ROUTING 


The  routing  of  the  MAE  is  planned  with  the  operational  objective  to  provide 
a  detection  coverage  that  is  as  extensive  as  possible  over  the  entire  area  of 
operation.  The  MAE  performs  the  classic  outward  spiral  search  pattern  by 
following  the  waypoints  as  depicted  in  Figure  A-1 .  Note  that  the  agents  in  MANA 
always  start  their  movement  from  the  waypoint  with  the  largest  index  and 
proceed  to  the  one  with  index  zero  in  a  reverse  numerical  order.  In  the  models, 
the  MAE  is  intended  to  loop  around  the  waypoints  such  that  it  flies  back  to  the 
starting  waypoint  once  the  last  one  is  reached. 


Figure  A-1  Routing  of  the  MAE 


70 


2. 


ONE-LAE  ROUTING 


In  the  One-LAE  configuration  shown  in  Figure  A-2,  the  one  and  only  LAE 
is  allocated  to  provide  surveillance  around  the  parameter  of  the  Red  objective, 
Twenty-nine  Palms,  where  critical  Red  activities  are  deemed  to  be  of  most 
interest  to  the  MEB  commander. 


Figure  A-2  Routing  of  One  LAE 
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3. 


TWO-LAE  ROUTINGS 


On  availability  of  a  second  LAE,  it  is  assigned  to  provide  coverage  over 
the  larger  of  the  two  cities,  Los  Angeles  and  its  suburbs,  while  leaving  the  first 
LAE  to  continue  its  surveillance  of  Twenty-nine  Palms.  The  only  intuitive 
justification  for  giving  priority  to  the  cities  over  the  forested  and  mountainous 
areas  when  allocating  areas  of  responsibilities  to  the  LAE  assets  is  based  on  the 
assumption  that  the  developed  areas  are  considered  to  be  more  accessible  by 
well-developed  road  networks  and  infrastructures  that  facilitate  deployment  of 
large  Red  forces.  The  routes  of  the  two  LAEs  are  shown  in  Figure  A-3. 


Figure  A-3  Routing  of  Two  LAEs 
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4. 


THREE-LAE  ROUTINGS 


With  the  third  LAE,  it  is  desired  to  extend  the  ISR  coverage  to  the  next 
larger  city,  San  Diego,  while  focusing  LAE  assets  on  the  regions  that  are  in  a 
direct  path  from  the  amphibious  landing  to  the  Red  objective.  The  routings  are 
shown  in  Figure  A-4.  The  area  of  coverage  may  appear  ambitious  for  a  local 
sensor  capability  as  an  LAE,  but  it  is  considered  important  to  establish  situational 
awareness  in  these  regions  where  many  Red  air  defense  elements  and  time 
critical  targets  are  expected  to  operate  to  defend  Twenty-nine  Palms  against  the 
Blue  attack. 


Figure  A-4  Routing  of  Three  LAEs 
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5. 


FOUR-LAE  ROUTINGS 


The  addition  of  the  fourth  LAE  provides  more  effective  ISR  coverage  of 
the  Los  Angeles  loop  whose  area  of  responsibility  is  initially  very  large  for  a 
single  LAE.  The  original  area  of  responsibility  is  now  broken  into  two  halves:  the 
western  and  eastern  loop,  each  allocated  to  an  LAE  as  illustrated  in  Figure  A-5. 


Figure  A-5  Routing  of  Four  LAEs 
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6. 


FIVE-LAE  ROUTINGS 


With  five  LAEs,  the  large  San  Diego  loop  is  now  divided  into  a  northern 
and  southern  loop  as  depicted  in  Figure  A-6.  A  single  LAE  is  responsible  for  the 
ISR  coverage  of  each  loop. 


Figure  A-6  Routing  of  Five  LAEs 
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APPENDIX  B.  LINEAR  REGRESSIONS,  PARTITIONING  TREES, 
GRAPHS  AND  PLOTS  FOR  ANALYSIS 


This  appendix  contains  the  linear  regression  outputs,  partitioning  trees, 
prediction  profiler  graphs  and  other  plots  used  in  the  complete  analytical  effort. 
The  materials  are  arranged  according  to  their  relevance  to  the  analytical  focus 
provided  by  the  initial  questions  of  interest. 


1.  SENSING  NETWORK  CONFIGURATIONS 

Distributions  of  the  proportion  of  total  red  targets  classified  for  one 
(leftmost)  to  five  (rightmost)  LAEs  show  a  general  trend  of  better  average 
classification  results  as  the  number  of  LAEs  increases.  Note  that  poor  results  can 
occur  for  all  situations. 
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Distributions  of  the  proportion  of  TCT  classified  for  one  (leftmost)  to  five 
(rightmost)  LAEs  show  a  general  trend  of  better  classification  results  as  the 
number  of  LAEs  increases.  Once  again,  poor  outcomes  can  occur  even  when  the 
number  of  LAEs  is  large. 
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Distributions  of  the  proportion  of  armor  classified  for  one  (leftmost)  to  five 
(rightmost)  LAEs  show  a  general  trend  of  better  classification  results  as  the 
number  of  LAEs  increases.  The  exception  observed  for  the  scenario  with  five 
LAEs  is  attributed  to  the  distribution  of  routes  when  the  fifth  LAE  is  introduced 
which  misses  most  of  the  armor  elements  in  Area  1  as  the  convoy  moves 
northwards. 
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99.5%  0.31475 

97.5%  0.25678 

90.0%  0.23150 

75.0%  quartile  0.20531 

50.0%  median  0.16350 

25.0%  quartile  0.10681 

10.0%  0.04350 

2.5%  0.02234 

0.5%  0.01087 

0.0%  minimum  0.01087 


Moments 


Mean  0.1527238 

Std  Dev  0.067695 

Std  Err  Mean  0.0059602 
upper  95%  Mean  0.1645171 
lower  95%  Mean  0.1409305 


Prop  Armor  Class 


Quantiles 


100.0%  maximum  0.42825 
99.5%  0.42825 

97.5%  0.41066 

90.0%  0.36075 

75.0%  quartile  0.31406 

50.0%  median  0.27337 

25.0%  quartile  0.18494 

10.0%  0.04400 

2.5%  0.03206 

0.5%  0.01050 

0.0%  minimum  0.01050 

Moments 

Mean  0.24197 

Std  Dev  0.1110752 

Std  Err  Mean  0.0097796 
upper  95%  Mean  0.2613206 
lower  95%  Mean  0.2226193 


Prop  Armor  Class 


Quantiles 


100.0%  maximum  0.47100 

99.5%  0.47100 

97.5%  0.45581 

90.0%  0.40950 

75.0%  quartile  0.36756 

50.0%  median  0.30650 

25.0%  quartile  0.21044 

10.0%  0.05050 

2.5%  0.02991 

0.5%  0.01038 

0.0%  minimum  0.01038 

Moments 

Mean  0.2716308 

Std  Dev  0.1271268 

Std  Err  Mean  0.0111929 
upper  95%  Mean  0.2937779 
low  er  95%  Mean  0.2494838 


99.5% 

97.5% 

90.0% 

75.0% 

50.0% 

25.0% 

10.0% 

2.5% 

0.5% 

0.0% 


quartile 

median 

quartile 


0.38138 

0.35828 

0.29938 

0.26363 

0.22075 

0.12350 

0.04825 

0.02119 

0.00775 

0.00775 


Moments 


Mean  0.1963246 

Std  Dev  0.0929332 

Std  Err  Mean  0.0081823 
upper  95%  Mean  0.2125147 
low  er  95%  Mean  0. 1 80 1 345 


N 


129 


N 


129 


N 


129 


N 


129 


N 


129 


The  bivariate  fits  for  the  proportion  of  target  classification  by  the  number  of 
LAEs  show  that  the  quadratic  terms  are  statistically  significant  in  the  models  for 
the  total  red  and  armor  targets  but  not  in  the  model  for  the  TCT  targets. 


Bivariate  Fit  of  Prop  Red  Class  By  #  LAE 


Polynomial  Fit  Degree=2 

Prop  Red  Class  =  0.1499145^0.0509699  #  LAE-  0.0212292  (#  LAE-3^2 

Summary  of  Fit 

RSquare  0.275092 

RSquare  Adj  0.272834 

Root  Mean  Square  Error  0.130754 

Mean  of  Response  0.260366 

Observations  (or  Sum  Wgts)  645 

Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

2 

4.165259 

2.08263 

121.8147 

Bror 

642 

10.976078 

0.01710 

Prob  >  F 

C.  Total 

644 

15.141337 

<.0001 

- Polynomial  Fit  Degree=2 


Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

R-ob>|t| 

Intercept 

0.1499145 

0.013552 

11.06 

<.0001 

#  LAE 

0.0509699 

0.003641 

14.00 

<.0001 

(#  LAE-3)A2 

-0.021229 

0.003077 

-6.90 

<.0001 
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Bivariate  Fit  of  Prop  TCT  Class  By  #  LAE 


0.3- 

0.25- 

8  0.2- 
O 

o  0.15- 

Q. 

e  o.i- 

Q. 

- 1— 

Polynomial  Fit  Degree=2 

i 

Prop  TCT  Class  =  0.0055754  +  0.01 98858  #  LAE  +  0.001 2658  (#  LAE-3  ^2 

Summary  of  Fit 

RSquare  0.238704 

RSquare  Adj  0.236333 

Root  Mean  Square  Error  0.050483 

Mean  of  Response  0.067764 

Observations  (or  Sum  Wgts)  645 

0- 

Analysis  of  Variance 

Source  DF  Sum  of  Squares  Mean  Square  F  Ratio 

Model  2  0.5130171  0.256509  100.6496 

Error  642  1.6361569  0.002549  Prob  >  F 

C.  Total  644  2.1491740  <.0001 

1  2  3  4  5 

#  LAE 

- Polynomial  Fit  Degree=2  | 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

0.0055754 

0.005232 

1.07 

0.2870 

#  LAE 

0.0198858 

0.001406 

14.15 

<.0001 

(#  LAE-3)A2 

0.0012658 

0.001188 

1.07 

0.2870 

Term  Estimate  Std  Error  t  Ratio  FTob>|t| 


Intercept  0.1443391 

#  LAE  0.0310841 

(#  LAE-3)A2  -0.022495 


0.009866  14.63  <.0001 

0.00265  11.73  <.0001 

0.00224  -10.04  <.0001 


A  stepwise  logistic  regression  model  is  fitted  for  the  categorical  response 
variable  lowTCTclass  to  investigate  the  rationale  for  high  frequencies  clustering 
around  the  low  value  as  depicted  in  the  distribution  graphs. 

if  proportion  of  TCT  classified  <  0.05 
otherwise 


lowTCTclass  = 
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Nominal  Logistic  Fit  for  Low  TCT  Class 
Whole  Model  Test 


Model  -LogLikelihood  DF  ChiSquare  Prob>ChiSq 

Difference  109.95633  6  219.9127  <0001 

Full  337.11662 

Reduced  447.07295 


RSquare  (U)  0.2459 

Observations  (or  Sum  Wgts)  645 
Converged  by  Gradient 

Lack  Of  Fit 


Source  DF 

Lack  Of  Fit  122 

Saturated  128 

Fitted  6 

-LogLikelihood 

104.84869 

232.26793 

337.11662 

ChiSquare 

209.6974 

Prob>ChiSq 

<.0001 

Parameter  Estimates 

1 

Term 

Estimate 

Std  Error  1 

ChiSquare 

Prob>ChiSq 

Intercept 

-4.0189294 

0.6504023 

38.18 

<.0001 

LAE  React  InOrg  SA 

0.48629692 

0.1619628 

9.02 

0.0027 

MAE  Detect  Range 

0.00019502 

0.0000467 

17.44 

<0001 

LAE  Class  Range 

0.00087507 

0.00009 

94.48 

<.0001 

LAE  Class  Prob 

1 .79752337 

0.4586052 

15.36 

<.0001 

TCT  Conceal 

-0.0241628 

0.0033894 

50.82 

<0001 

TCT  Detect  Range 

For  log  odds  of  0/1 

-0.138945 

0.0320138 

18.84 

<0001 

The  model  reveals  that  the  factors  of  significance  for  the  low  classification 
rates  are  LAE  classification  range  and  classification  probability,  LAE 
responsiveness  to  other  UAV  SA,  MAE  detection  range,  TCT  concealment  and 
TCT  evasion  detection  range. 

The  prediction  profiler  graph  for  the  logistic  regression  model  suggests 
that  the  LAE  classification  range  is  the  most  influential  factor  given  that  all  others 
are  kept  constant  at  their  nominal  values. 


T-  T— 

CM 

CO 

CO 

CO  t- 

o  O 

o  O 

'  0.00031 

h» 

CO 

8454.64  S 

CO 

3600  ° 

o  0.65039 

50.0155  ° 

5.08527  T" 

CM 

CO 

LAE  React 

MAE  Detect 

LAE  Class 

LAE  Class 

TCT  Conceal 

TCT  Detect 

InOrg  SA 

Range 

Range 

FTob 

Range 
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The  partitioning  tree  for  the  proportion  of  TCT  targets  further  suggests  that 
an  LAE  with  classification  range  less  than  approximately  3,200  meters  has  a 
mean  classification  proportion  of  only  0.04.  Even  if  the  LAE  has  better 
classification  capabilities,  the  TCT  classification  rate  is  even  worse  if  the  TCT 
concealment  is  above  85%. 


A  similar  analytical  approach  is  also  conducted  to  investigate  the  rationale 
for  the  low  proportion  of  armor  classified.  The  categorical  response  variable 
lowARMORclass  is  defined  as  follows. 

[T  if  proportion  of  armor  classified  <  0. 1 
lowARMORclass  =  l 

0  otherwise 
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Nominal  Logistic  Fit  for  Low  Armor  Class 

Whole  Model  Test 

Model 

-LogLikelihood 

DF 

ChiSquare 

F’robXDhiSq 

Difference 

197.73733 

4 

395.4747 

<.0001 

Full 

166.87878 

Reduced 

364.61611 

RSquare  (U) 

0.5423 

Observations  (or  Sum  Wgts^ 

)  645 

Converged  by  Gradient 

[Lack  Of  Fit 

i 

Source 

DF  -LogLikelihood 

ChiSquare 

Lack  Of  Fit 

124 

73.35416 

146.7083 

Saturated 

128 

93.52462 

Prob>ChiSq 

Fitted 

4  166.87878 

0.0802 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

ChiSquare 

Rob>ChiSq 

Intercept 

-10.494934 

1 .089704 

92.76 

<.0001 

LAE  React  InOrg  SA 

-0.5623604 

0.2333648 

5.81 

0.0160 

MAE  Class  Range 

0.00027273 

0.0000733 

13.86 

0.0002 

LAE  Class  Range 

0.00267633 

0.0002349 

129.79 

<0001 

MAE  Class  [Rob 

5.50551346 

1 .0309848 

28.52 

<.0001 

For  log  odds  of  0/1 


The  model  suggests  that  LAE  classification  range,  LAE  responsiveness  to 
other  UAV  SA,  MAE  classification  range  and  classification  probability  are 
significant  determinants  of  the  low  classification  rates  for  armor  targets. 


The  LAE  classification  range  is  again  identified  by  the  prediction  profiler  as 
the  most  significant  factor  given  that  all  others  are  kept  at  their  nominal  levels. 


0.00031  °  6784.13  £  ”  3569.77°  JE-  0.35775° 

CO  T-  T-  CO  ° 

LAE  React  MAE  Class  LAE  Class  MAE  Class 
InOrg  SA  Range  Range  Rob 


The  partitioning  tree  shows  that  an  LAE  with  classification  range  of  less 
than  2,600  meters  has  a  mean  proportion  of  armor  targets  classified  of  about 
0.06. 


83 


R-square  =  0.494 


Count  645 

Mean  0.1926014 
StdDev  0.1113035 


LAE  Class  Range<2579 

Count 

145 

Mean 

0.0576078 

Std  Dev 

0.0325195 

LAE  Class  Range>=2579 

Count  500 

Mean  0.2317495 
Std  Dev  0.094086 


MAE  Class  Range<6304 

MAE  Class  Range>=6304 

LAE  Class  Range<2866 

Count  65 

Mean  0.0348596 

Std  Dev  0.0140108 

Count  80 

Mean  0.0760906 

Std  Dev  0.0315735 

Count  45 

Mean  0.1548417 

Std  Dev  0.063788 

LAE  Class  Range>=2866 

Count  455 

Mean  0.2393558 
Std  Dev  0.0932255 


MAE  Class  Prob<0.32 


Count  205 

Mean  0.2175262 
Std  Dev  0.0879724 


MAE  Class  Prob>=0.32 

Count  250 

Mean  0.257256 

Std  Dev  0.093764 


2.  FACTORS  OF  SIGNIFICANCE 

Next,  Multiple  Linear  Regression  models  are  provided  for  the  proportion  of 
TCT  classified  with  consideration  of  only  the  main  effects  (top  left),  the  main  and 
quadratic  effects  (top  right)  and  including  the  two-way  interactions  (bottom). 


Response  Prop  TCT  Class 


Response  Prop  TCT  Class 


Summary  of  Fit 


Summary  of  Fit 


.**  0.645179 


*  0.576524 


RSquare 

RSquare  Adj  _ _  , 

Root  Mean  Square  Error 

Mean  of  Response  0.067764 


RSquare 
RSquare  Adj 
Root  Mean  Square  Error 
Mean  of  Response  0.067764 

Observations  (or  Sum  Wgts)  645 

Analysis  of  Variance 


Observations  (or  Sum  Wgts) 

645 

DF 

Sum  of  Squares 
1.3972599 

Mean  Square 
0.155251 

Analysis  of  Variance 

1 

Model 

9 

131.1113 

Source  IDF  Sum  of  Squares 

Mean  Square  F  Ratio 

Error 

635 

0.7519141 

0.001184 

R-ob>  F 

Model  7  1.2489432 

0.178420  126.2497 

C.  Total 

644 

2.1491740 

<.0001 

Error  637 

0.9002308 

0.001413 

Prob  >  F 

Parameter  Estimates 

1 

C.  Total  644 

2.1491740 

<0001 

Term 

Intercept 

Estimate 

-0.06237 

Std  Error 

0.008775 

t  Ratio 

-7.11 

Prob>|t| 

<0001 

Parameter  Estimates 

HI 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

LAE  React  InOrg  SA 

0.0117325 

0.00233 

5.04 

<0001 

Intercept 

-0.07178 

0.009439 

-7.60 

<.0001 

MAE  Class  Range 

0.000004 

6.888e-7 

5.84 

<0001 

LAE  React  InOrg  SA 

0.011604 

0.002545 

4.56 

<.0001 

LAE  Class  Range 

0.0000212 

0.000001 

19.46 

<.0001 

MAE  Class  Range 

0.0000039 

7.521  e-7 

5.24 

<.0001 

LAE  Class  Prob 

0.0503286 

0.006507 

7.74 

<0001 

LAE  Class  Range 

0.0000185 

0.000001 

16.12 

<.0001 

TCT  Conceal 

-0.000549 

0.000047 

-11.78 

<0001 

LAE  Class  Prob 

0.0492387 

0.007107 

6.93 

<.0001 

TCT  Detect  Range 

-0.003587 

0.000451 

-7.96 

<0001 

TCT  Conceal 

-0.000533 

0.000051 

-10.47 

<.0001 

#  LAE 

0.0198858 

0.000958 

20.76 

<0001 

TCT  Detect  Range 

-0.003614 

0.000492 

-7.34 

<.0001 

(LAE  Class  Range-3569.77)*(LAE  Class  Range-3569.77) 

-6.907e-9 

7.86e-10 

-8.79 

<0001 

#  LAE 

0.0198858 

0.001047 

19.00 

<.0001 

(TCT  Conceal-50.01 55)*(TCT  Conceal-50.0155) 

-0.000009 

0.000002 

-5.19 

<0001 
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Response  Prop  TCT  Class 
Summary  of  Fit 

RSquare 

RSquare  Adj  {  0.725647  %l 

Root  Mean  Square  Error  *0.080 3£&* 

Mean  of  Response  0.067764 

Observations  (or  Sum  Wgts)  645 

Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

14 

1.5723599 

0.112311 

122.6672 

Error 

630 

0.5768141 

0.000916 

Prob  >  F 

C.  Total 

644 

2.1491740 

<.0001 

Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

-0.062471 

0.007717 

-8.10 

<.0001 

LAE  React  InOrg  SA 

0.0120452 

0.002049 

5.88 

<.0001 

MAE  Class  Range 

0.0000039 

6.059e-7 

6.50 

<.0001 

LAE  Class  Range 

0.0000212 

9.586e-7 

22.09 

<.0001 

LAE  Class  Rob 

0.0503922 

0.005722 

8.81 

<.0001 

TCT  Conceal 

-0.000549 

0.000041 

-13.40 

<.0001 

TCT  Detect  Range 

-0.003615 

0.000397 

-9.11 

<.0001 

#  LAE 

0.0198858 

0.000842 

23.60 

<.0001 

(LAE  React  InOrg  SA-0.00031)*(LAE  Class  Range-3569.77) 

0.0000076 

0.000002 

4.38 

<.0001 

(LAE  Class  Range-3569.77)*(#  LAE-3) 

0.000006 

6.537e-7 

9.19 

<.0001 

(LAE  Class  Prob-0.65039)*(#  LAE-3) 

0.0137356 

0.004044 

3.40 

0.0007 

(TCT  Conceal-50.01 55)*(TCT  Detect  Range-5.08527) 

-0.000093 

0.000016 

-5.85 

<.0001 

(TCT  Conceal-50.01 55)*(#  LAE-3) 

-0.000156 

0.000029 

-5.39 

<.0001 

(LAE  Class  Range-3569. 77)*(LAE  Class  Range-3569.77) 

-7.01 3e-9 

7e-10 

-10.02 

<.0001 

(TCT  Conceal-50.01 55)*(TCT  Conceal-50.01 55) 

-0.000008 

0.000002 

-5.05 

<.0001 

Note  that  the  ratios  used  in  the  experimental  design  and  their  basic 
parameters  (for  example,  ratio  of  MAE/LAE  classification  probabilities,  MAE 
classification  probability,  and  LAE  classification  probability)  are  all  included  as 
potential  terms  in  the  model  selection  process.  However,  none  of  the  ratios 
appeared  in  any  of  the  stepwise  models,  while  some  of  the  basic  parameters  (for 
example,  LAE  classification  probability)  are  included  in  the  selected  model. 

The  adjusted  R-squares  as  opposed  to  the  standard  R-squares  for  the 
three  models  are  considered  when  selecting  the  appropriate  base  model  for 
analysis  so  that  the  matrices  are  normalized  against  the  number  of  terms  used  in 
the  models.  The  full  model  (includes  the  quadratic  and  interaction  terms)  explains 
73%  of  the  variability,  while  the  main  effect  model  and  quadratic  model  explain 
only  58%  and  65%  respectively.  The  quadratic  and  interaction  terms  account  for 
about  20%  of  the  explanatory  power  in  the  full  model.  These  terms  are  therefore 
deemed  important  and  the  full  model  is  used  a  baseline  model  for  analysis. 
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The  actual  by  predicted  plot  on  the  left  shows  the  lower  tail  of  the 
distribution  being  cut  off  as  the  actual  proportion  can  only  take  on  values 
between  zero  and  one.  Similarly,  there  are  no  negative  residuals  when  the 
predicted  value  is  less  than  or  equal  zero,  hence  the  residual  by  predicted  plot 
appears  capped  by  the  45  degree  boundary  on  the  bottom  left  of  the  plot.  This 
departure  from  the  regression  assumption  of  additive  constant  variance  noise 
would  be  of  more  concern  if  we  were  interested  in  making  accurate  numerical 
predictions  of  classification  capabilities,  rather  than  in  identifying  those  factors 
that  have  the  greatest  impact  on  classification  capability.  It  also  shows  the  value 
of  obtaining  qualitatively  similar  results  from  several  different  analysis  methods, 
such  as  partitioning  trees,  Multiple  Linear  Regression,  and  logistic  regression. 


The  Pareto  plot  displays  the  factors  of  significance  in  descending  order  in 
accordance  to  their  effects  on  the  proportion  of  TCT  classified. 


Pareto  Plot  of  Transformed  Estimates 


Term 

Orthog  Estimate 

#  LAE 

0.0281228 

LAE  Class  Range 

0.0240802 

TCT  Conceal 

-0.0155919 

(LAE  Class  Range-3569.77)*(LAE  Class  Range-3569.77) 

-0.0134044 

(LAE  Class  Range-3569.77)*(#  LAE-3) 

0.0110785 

TCT  Detect  Range 

-0.0108626 

LAE  Qass  FVob 

0.0104792 

(TCT  Conceal-50.01 55)*(TCT  Detect  Range-5.08527) 

-0.0082651 

LAE  React  InOrg  SA 

0.0068911 

MAE  Class  Range 

0.0068698 

(TCT  Conceal-50.01 55)*(#  LAE-3) 

-0.0064212 

(TCT  Conceal-50.01 55)*(TCT  Conceal-50.01 55) 

-0.0060193 

(LAE  React  InOrg  SA-0.00031)*(LAE  Class  Range-3569.77) 

0.0060000 

L 

(LAE  Qass  Prob-0.65039)*(#  LAE-3) 

0.0041515 
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The  prediction  profiler  graph  shows  a  significant  increasing  linear 
relationship  between  the  response  variable  and  the  number  of  LAEs,  which  also 
appears  in  the  Pareto  plot  as  the  factor  with  the  greatest  impact.  The  graph  also 
illustrates  the  non-linear  relationships  of  the  response  variable  with  both  LAE 
classification  range  and  TCT  concealment  ability. 


Prediction  Profiler 

l_  0.31733  “ 

P  CO 

*7  $  0.086441 
o  o  ±0.0039 

Q. 

o  - 

0.00031  2  6784.13  £  5  3569.77  8  °  0.65039  50.0155  2  5.08527^  3 

CO  T-  T-  CD 

LAE  React  MAE  Class  LAE  Class  LAE  Class  TCT  Conceal  TCT  Detect  #  LAE 

InOrg  SA  Range  Range  Prob  Range 


The  interaction  plot  highlights  the  statistically  significant  interactions  in 
solid  lines. 


-1  0  1  2000  4000  6000  .3.4.5.6.7.8.91  0  20406080  110  0123456789  11  1  2  3  4  5 
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The  partitioning  tree  shows  a  partition  of  the  mean  proportion  of  TCT 
classification  that  explains  approximately  68.5%  of  the  response  variability.  The 
right  branch  of  each  partition  represents  the  split  that  offers  a  higher  mean 
classification  rate  than  the  left  branch.  The  first  few  tiers  of  the  tree  depict  the 
most  significant  factors.  The  results  are  consistent  with  those  suggested  by  the 
earlier  observations. 


R-square  =  0.685 


LAE  Detect  Range<4585 

Count 

192 

Mean 

0.0293715 

Std  Dev 

0.0197051 

LAE  Detect  Range>=4585 

Count 

195 

IVtean 

0.0596342 

Std  Dev 

0.0397148 

TCT  Conceal>=88 


TCT  Conceal<88 

Count  165 

Mean  0.0673374 
Std  Dev  0.0379217 


LAE  Class  Range<2866 

Count  76 

Mean  0.0378333 
Std  Dev  0.0284235 


TCT  Conceal>=86 


LAE  Clas 

Range>=2866 

Count 

182 

Mean 

M294762 

Std  Dev 

0.0599873 

TCT  Conceal<86 


#  LAE<2 


LAE  Class  Range<3439 

LAE  Class  Range>=3439 

Count  38 

Mean  0.1067719 

Std  Dev  0.0332393 

Count  118 

Mean  0.1554689 

Std  Dev  0.0509618 

The  Multiple  Linear  Regression  models  are  also  provided  for  the 
proportion  of  armor  classified  with  consideration  of  only  the  main  effects  (left),  the 
main  and  quadratic  effects  (right)  and  including  the  two-way  interactions  (next 
page). 
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Response  Prop  Armor  Class 


Response  Prop  Armor  Class 
Summary  of  Fit 

RSquare  0.546679 

RSquare  Adj 

Root  Mean  Square  Error  0.07535  J> 
Mean  of  Response 

Observations  (or  Sum  Wgts)  645 

Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

7 

4.3614987 

0.623071 

109.7406 

Error 

637 

3.6166781 

0.005678 

Prob>F 

C.  Total 

644 

7.9781769 

<0001 

Summary  of  Fit 

RSquare  0.777112 

RSquare  Adj 

Root  Mean  Square  Error  0.052919  / 

Mean  of  Response  T)Tf92WTf 

Observations  (or  Sum  Wgts)  645 

Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

9 

6.1999396 

0.688882 

245.9965 

Error 

635 

1.7782373 

0.002800 

Rob  >  F 

C.  Total 

644 

7.9781769 

<0001 

Parameter  Estimates 


Parameter  Estimates 

Term 

Intercept 

Estimate 

-0.218018 

Std  Error 

0.019501 

t  Ratio 

-11.18 

Prob>|t| 

<.0001 

Term 

Estimate 

Std  Error 

t  Ratio 

FTob>|t| 

LAE  React  InOrg  SA 

-0.014634 

0.003583 

-4.08 

<.0001 

Intercept 

-0.282286 

0.028265 

-9.99 

<0001 

LAE  React  En  Contact 

0.0154657 

0.003583 

4.32 

<.0001 

LAE  React  En  Contact 

0.0175678 

0.005101 

3.44 

0.0006 

LAE  Airspeed 

0.000569 

0.000088 

6.45 

<.0001 

LAE  Airspeed 

0.000538 

0.000126 

4.28 

<0001 

MAE  Class  Range 

0.0000054 

0.000001 

5.14 

<.0001 

MAE  Detect  Range 

0.0000049 

0.000001 

3.42 

0.0007 

LAE  Class  Range 

0.0000567 

0.000002 

33.85 

<.0001 

LAE  Detect  Range 

0.0000168 

0.000005 

3.61 

0.0003 

LAE  Class  Prob 

0.0964538 

0.010006 

9.64 

<.0001 

LAE  Class  Range 

0.000032 

0.000005 

6.25 

<.0001 

#  LAE 

0.0310841 

0.001473 

21.10 

<.0001 

LAE  Class  FTob 

0.0926305 

0.01425 

6.50 

<0001 

(LAE  Class  Range-3569. 77)*(LAE  Class  Range-3569.77) 

-2.196e-8 

1.187e-9 

-18.49 

<.0001 

#  LAE 

0.0310841 

0.002098 

14.82 

<.0001 

(#  LAE-3)*(#  LAE-3) 

-0.022495 

0.001245 

-18.06 

<.0001 

Response  Prop  Armor  Class 
Summary  of  Fit 

RSquare 

RSquare  Adj  0.804814  1 

Root  Mean  Square  Error  *07^49774 

Mean  of  Response  0.192601 

Observations  (or  Sum  Wgts)  645 

Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

12 

6.4499617 

0.537497 

222.2841 

Error 

632 

1.5282152 

0.002418 

FTob  >  F 

C.  Total 

644 

7.9781769 

<.0001 

Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  F?atio 

Rob>|t| 

Intercept 

-0.221601 

0.018155 

-12.21 

<.0001 

LAE  React  InOrg  SA 

-0.014474 

0.00333 

-4.35 

<.0001 

LAE  React  En  Contact 

0.0158282 

0.003331 

4.75 

<.0001 

LAE  Airspeed 

0.0005793 

0.000082 

7.06 

<.0001 

MAE  Class  Flange 

0.0000056 

9.857e-7 

5.72 

<.0001 

LAE  Class  Range 

0.0000568 

0.000002 

36.48 

<.0001 

MAE  Class  FTob 

0.0617215 

0.01893 

3.26 

0.0012 

LAE  Class  Rob 

0.0626223 

0.013932 

4.49 

<.0001 

#  LAE 

0.0310841 

0.001369 

22.70 

<.0001 

(LAE  Airspeed-1 70. 109)*(#  LAE-3) 

0.0002518 

0.000058 

4.34 

<.0001 

(LAE Class  Range-3569.77)*(#  LAE-3) 

0.0000091 

0.000001 

8.58 

<.0001 

(LAE  Class  F?ange-3569.77)*(LAE  Class  Range-3569.77) 

-2.191e-8 

1.103e-9 

-19.85 

<.0001 

(#  LAE-3)*(#  LAE-3) 

-0.022495 

0.001157 

-19.44 

<.0001 

As  before,  the  full  model  explains  80%  of  the  variability  while  the  main 
effect  model  and  quadratic  model  explain  only  54%  and  77%  respectively.  The 
interaction  terms  account  for  less  than  4%  of  the  explanatory  power  in  the  full 
model.  These  terms  are  therefore  deemed  insignificant  and  the  quadratic  model 
is  the  model  of  choice  for  further  analysis. 
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The  actual  by  predicted  plot  on  the  left  shows  the  lower  tail  of  the 
distribution  being  cut  off  as  the  actual  proportion  can  only  take  on  values 
between  zero  and  one.  In  the  same  manner,  there  are  no  negative  residuals 
when  the  predicted  value  is  less  than  or  equal  to  zero,  hence  the  residual  by 
predicted  plot  appears  capped  by  the  45  degree  boundary  on  the  bottom  left  of 
the  plot.  These  results  are  less  pronounced  that  for  the  TCT  classification  model. 
There  is  also  some  evidence  of  nonlinearity,  which  may  be  due  to  the  omission  of 
interaction  terms,  or  because  the  underlying  relationships  that  reveal  diminishing 
returns  are  not  exactly  quadratic. 


Residual  by  Predicted  Plot 
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The  Pareto  plot  displays  the  factors  of  significance  in  descending  order  in 
accordance  to  their  effects  on  the  proportion  of  armor  targets  classified. 


Pareto  Plot  of  Transformed  Estimates 


Term 

Orthog  Estimate 

LAE  Class  Range 

0.0631298 

#  LAE 

0.0439596 

(LAE  Class  Range-3569.77)*(LAE  Class  Range-3569.77) 

-0.0385286 

(#  LAE-3)*(#  LAE-3) 

-0.0376414 

LAE  Class  Prob 

0.0192751 

LAE  Airspeed 

0.0134963 

LAE  React  En  Contact 

0.0104379 

LAE  React  InOrg  SA 

-0.0086792 

MAE  Class  Range 

0.0074305 
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The  prediction  profiler  graph  shows  significant  non-linear  relationships  of 
the  response  variable  against  the  number  of  LAEs  and  LAE  classification  range, 
both  of  these  factors  and  their  quadratics  appear  in  the  Pareto  plot  as  the  highest 
four  factors  with  the  most  impact  on  the  proportion  of  armor  targets  classified. 


Prediction  Profiler 
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InOrg  SA  En  Contact  Airspeed  Range  Range  Prob 


The  partitioning  tree  depicts  a  partition  of  the  mean  proportion  of  armor 
classification  which  explains  approximately  83.6%  of  the  data  variability.  The  first 
few  tiers  of  the  tree  capture  the  non-linear  relationships  of  the  classification  rate 
against  the  number  of  LAEs  and  LAE  classification  range  as  well. 


LAE  Class  Range<2579 

Count 

145 

Mean 

0.0576078 

Std  Dev 

0.0325195 

#  LAE<3 


All  Rows 

Count  645 

Mean  0.1926014 
StdDev  0.1113035 


I 

LAE  Class  Range>=2579 

Count  500 

Mean  0.2317495 
Std  Dev  0.094086 


#  LAE>=3 


R-square  =  0.836 


Count  200 
Mean  0.1487369 
Std  Dev  0.04971 


Count  300 
Mean  0.2870913 
Std  Dev  0.0737993 


#  LAE<2 

#  LAE>=2 

#  LAE>=5 

#  LAE<5 

Count  100 

Mean  0.1156525 

Std  Dev  0.0297929 

Count  100 

Mean  0.1818212 

Std  Dev  0.0432232 

Count  100 

Mean  0.236255 

StdDev  0.0611649 

Count  200 

Mean  0.3125094 

Std  Dev  0.0661365 

LAE  Airspeed<166 

LAE  Airspeed>=166 

LAE  Class  Range<3439 

LAE  Class  Range>=3439 

Count  42 

Mean  0.1945208 

Std  Dev  0.0459626 

Count  58 

Mean  0.2664763 

Std  Dev  0.0525878 

Count  62 

Mean  0.2615948 

Std  Dev  0.053543 

Count  1 38 

Mean  0.3353841 

StdDev  0.0581103 
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3. 


UAV  BEHAVIORS 


Contour  plots  showing  the  responsiveness  of  the  LAEs  toward  newly 
classified  enemy  contacts  and  MAE  redirections  with  respect  to  following  the  pre¬ 
assigned  routings  to  support  high  target  classification  proportions. 


The  dark  blue  patches  show  higher  proportions  of  target  classification  than 
the  light  blue  patches.  It  should  be  noted  that  the  scales  for  the  two  contour  plots 
are  different.  The  plots  show  generally  higher  classification  proportions  on  TCTs 
if  an  LAE  is  more  reactive  to  MAE  redirection,  while  no  obvious  benefit  is  seen  by 
having  an  LAE  be  more  reactive  to  a  new  classified  enemy  contact.  For 
classifying  armor  targets,  having  LAEs  react  less  to  MAE  redirection  and  more  to 
a  new  classified  enemy  contact  seems  to  improve  the  overall  classification 
performance. 
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4. 


IMPACT  OF  LAE  FAILURE 


The  following  graphs  depict  the  average  proportions  of  TCT  and  armor 
classification  due  to  the  failure  of  an  LAE  at  various  time  instances  in  the 
scenario  with  and  without  cooperative  sensing  capability. 


TCT  Classification  Proportion  Against  LAE  Lifetime  With  and 
Without  Cooperative  Sensing 


-♦ —  No  Cooperation 
■m —  Cooperation 


Armor  Classification  Proportion  Against  LAE  Lifetime  With 
and  Without  Cooperative  Sensing 


■  No  Cooperation 

■  Cooperation 
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